In ordinary calculus, you define “integral derivatives” — the first derivative, second derivative, etcetera. If you think of differentiation as an operator D that takes functions to functions, then the higher-order derivatives are just Di for natural numbers i. As far back as Liouville, mathematicians have defined fractional derivatives, extensions of this definition to real numbers, i. There is more than one possible definition, Wikipedia page gives the usual definition, which is in terms of the Laplace transform.
I’ve never known if fractional derivatives were good for anything, or were just a historical curiosity. (They are a special case of singular integral operators, which are useful in PDEs.) This very brief paper discusses an application of fractional derivatives to models of particles in a liquid. This sounds like it should be related to Brownian motion, and it is, but the processes that arise are related to more general LÃ©vy processes than just Brownian motion.