Stick-Slip of Box on a Moving Belt

The motion of the belt under the block and the friction between the two complicates the issue. Once there is relative velocity between the block and the belt your equation comes into play to describe the motion of the block relative to the spring attachment point. Here I have a difficulty. Since x is taken to be the displacement from the attachment point, then x' must be the velocity relative to that point, not the velocity relative to the belt. In that case the belt velocity does not appear explicitly in your equation.

Let's look at the startup situation. Initially the block and the belt are motionless and the spring is relaxed. In this situation the coefficient of friction is the static coefficient. As the belt begins to move, the block moves with it until the tension in the spring is sufficient to break the static friction. At this point the frictional force is the normal force times the kinetic coefficient of friction which is smaller than the static coefficient. The spring force then exceeds the force of kinetic friction, drawing the block back toward the attachment point of the spring. I anticipate some sort of oscillatory motion

If we re-write your equation a bit so that:
x''=-x-x'(bx'^{2}-a) we have a
simple harmonic oscillator with an energy loss when
bx'^{2} greater than a and an energy gain when
bx'^{2} less than a. For small x', out near the
ends of the oscillatory motion, the belt couples energy into the
system. Near the center of the oscillatory motion the block
looses energy to the belt. That is what your equation
suggests.

This leaves me with the fundamental problem that the energy loss-gain seems not to depend on the velocity of the belt.

Please let me know if I have misinterpreted the situation or the equation. I would be happy to discuss it further if these thoughts prompt any questions from you.

The first term is eliminated from the equation of motion by
this substitution. The second and the fourth terms are somehow
present with a little imagination. What about the third term in
such explanation. Maybe this explanation is not the most
appropriate. I'd just like to know from where all the terms
come. Otherwise we can say - the equilibrium point is unstable
and we have a repellor. Therefore we need negative dumping for
small absolute values of the velocity of the mass and positive
dumping for higher velocities. How can we get that? By applying
combined damping -ax'+b(x')^{3}. The result is
limit cycle.

The expression which we are left with does generate a limit cycle, driving large oscillations down and small oscillations up for a and b greater than 0. I ran it through DynaLab-Pro. The vector field view is seen below. The green line is the velocity nullcline. The dark blue line is the acceleration nullcline, the red track is the system trajectory in phase space starting at (1.5,0). The light blue track is the trajectory starting at (0.11,0). We see an unstable spiral fixed point at the origin enclosed by the limit cycle on which the trajectories settle out. Hope this helps.