The motion of the belt under the block and the friction between the two complicates the issue. Once there is relative velocity between the block and the belt your equation comes into play to describe the motion of the block relative to the spring attachment point. Here I have a difficulty. Since x is taken to be the displacement from the attachment point, then x' must be the velocity relative to that point, not the velocity relative to the belt. In that case the belt velocity does not appear explicitly in your equation.
Let's look at the startup situation. Initially the block and the belt are motionless and the spring is relaxed. In this situation the coefficient of friction is the static coefficient. As the belt begins to move, the block moves with it until the tension in the spring is sufficient to break the static friction. At this point the frictional force is the normal force times the kinetic coefficient of friction which is smaller than the static coefficient. The spring force then exceeds the force of kinetic friction, drawing the block back toward the attachment point of the spring. I anticipate some sort of oscillatory motion
If we re-write your equation a bit so that: x''=-x-x'(bx'2-a) we have a simple harmonic oscillator with an energy loss when bx'2 greater than a and an energy gain when bx'2 less than a. For small x', out near the ends of the oscillatory motion, the belt couples energy into the system. Near the center of the oscillatory motion the block looses energy to the belt. That is what your equation suggests.
This leaves me with the fundamental problem that the energy loss-gain seems not to depend on the velocity of the belt.
Please let me know if I have misinterpreted the situation or the equation. I would be happy to discuss it further if these thoughts prompt any questions from you.
The first term is eliminated from the equation of motion by this substitution. The second and the fourth terms are somehow present with a little imagination. What about the third term in such explanation. Maybe this explanation is not the most appropriate. I'd just like to know from where all the terms come. Otherwise we can say - the equilibrium point is unstable and we have a repellor. Therefore we need negative dumping for small absolute values of the velocity of the mass and positive dumping for higher velocities. How can we get that? By applying combined damping -ax'+b(x')3. The result is limit cycle.
The expression which we are left with does generate a limit cycle, driving large oscillations down and small oscillations up for a and b greater than 0. I ran it through DynaLab-Pro. The vector field view is seen below. The green line is the velocity nullcline. The dark blue line is the acceleration nullcline, the red track is the system trajectory in phase space starting at (1.5,0). The light blue track is the trajectory starting at (0.11,0). We see an unstable spiral fixed point at the origin enclosed by the limit cycle on which the trajectories settle out. Hope this helps.