This is more apparent when using an AJAX refreshless search method, but affects all joomla sites using the default search settings. Here's how to speed it up...
The default installation of joomla comes with all search plugins enabled. Standard plugins include:
I don't use the joomla weblinks system, so why do I want it to be searchable?
The same goes for newsfeeds and contacts. They can all go off!
Which leaves us with content,categories and sections.
If your site uses section and category views you may wish to keep them enabled. This site uses the category view but currently has no information other than article titles. So out of 6 plugins I really only need 1 enabled!
To Turn Off Joomla Search Plugins
Click on extensions then Plugin Manager, this displays all plugins. Use the filter top-right to list the type search, you can now disable the plugins you don't need.
This cuts down the calls to the database and produces faster results.
If you use joomfish there will be the same plugins starting with jf eg. search-jfsections
Search speed can also be affected by your php memory limit, see here.