Why to write the whole thing from the scratch if the same thing is already available ? Better to build on top! Isn’t it ?
Framework says : Concentrate on your objective rather then supporting things needed to accomplish the objective. Framework provide the facility and tools and expect the programmer to build the things on top of it.
crawler4j is a framework to that provide multi-threaded crawler with some extra features like Logging, customization’s, crawling password protected sites etc.
Here is a small example of starting with crawler4j.
Build env :Maven