This will make one side sharp, tapering all the way from the non-sharp side, like a real knife.
Apply the Solidify modifier
Merge the two edge rows on the sharp end of the blade
- Edit mode
- Edge select mode
- CTRL+TAB > Edge
- Select edges on sharp end of blade
- S > X > 0
- Replace X with whatever axis is perpendicular to the flat side of blade
- A (once or twice to select all)
- W > Remove Doubles
- Optionally, you can manually scale the edges on the other side to taper as desired (see picture)
This will only work if your knife is oriented straight along some axis. See here for more details on merging edge loops.