how to get rid of xdebug cannot be loaded error

how to get rid of xdebug cannot be loaded error

How do I get rid off this error when I run php-fpm
In my phpinfo, xdebug is mentioned, so why do I get this error
Failed loading xdebug.so: dlopen(xdebug.so, 9): image not found
I have php54 installed with Homebrew if that matters.
in php-fpm log
Failed loading xdebug.so: dlopen(xdebug.so, 9): image not found
thanks