If you are serious about making gains, I see two things you need: a good idea of your maintenance calories and a good routine.
If you aren't already tracking your calories and your macros, it's the FIRST thing you should do. How can you clean bulk (which is where you gain maximum muscle and minimum fat) if you don't know how many calories you need to do so?
Download a calorie tracker app if you don't already have one (I use MyFitnessPal--it's free). It'll have you input your age, height, and weight. Then it'll tell you roughly how many calories you should eat a day. For macros, aim for 150g protein and split the rest 50/50 between carbs and fat.
For a routine, I'd recommend a 5x5 program like starting strength. It may seem silly dropping your bench back down to 45lb (it's where you start), but the weight will go up FAST. It focuses on building a foundation of strength, but you'll gain muscle too. Once you get to where you can bench your body weight for reps and squat 1.5x your body weight for reps, you'll be ready to make the most of s hypertrophy program (PHAT is good and popular).
Back to the calorie tracking. Your first week, set your calories to maintain weight. The app is a best guess, so you need to hone that in. If you lose weight, go up 200 calories and try again for a week. If you gain, drop 200. Once you have a week where your weight doesn't change, you know your maintenance levels. From there, you can bulk or cut. I've had great luck doing 8 week lean bulks (+250 calories) and 4 week cuts (-500 calories). For me that cycle minimizes fat gains (bulking) and muscle losses (cutting).
I started with a build very similar to yours and spent nearly two years making little progress because I did what you currently are, guessed at my calories and didn't follow a solid routine (I even logged my lifts like you're doing). So I'm suggesting all this being just a bit further down a similar path. So I hope you find it helpful.