A new multi-search engine for querying data through an Internet search service on CORBA Yue-Shan Chang a, * , Shyan-Ming Yuan a , Winston Lo b a Department of Computer and Information Science, National Chiao Tung University, 1001 TA Hsueh Road, Hsin-Chu 30050, Taiwan, ROC b Department of Computer and Information Science, Tung Hai University, Taichung, Taiwan, ROC Received 14 February 2000; received in revised form 5 May 2000; accepted 19 May 2000 Abstract Search engines are important but generally far from ideal tools of the World Wide Web (WWW). Many researchers therefore prefer to use meta-brokers to construct multi-search engines (MSE). However, these have no uniform pro- gramming interfaces, which makes tying them with other search engines dicult. Moreover, for an application that needs a search service capability, querying them is dicult. To reduce that diculty, we propose in this paper an Internet search service (ISS) based on common object request broker architecture (CORBA) that follows the style of common object service specification (COSS). We design a multi- search engine based on ISS, which we term Octopus. For a system developer, because of its uniformity of interface, Octopus easily ties with any search engine. Equally, for an application programmer, the ISS oers a clear interface for application programs to search for information or mine data from the Internet. We demonstrate our approach to designing multi-search engines through ISS by tying two search engine agents, Yahoo and AltaVista, with Octopus and show how CORBA clients query them. Programmers may use this interface to construct their search engine agents or query a search engine agent in their applications. Ó 2000 Elsevier Science B.V. All rights reserved. Keywords: Internet; Multi-search service; Search engine; World Wide Web; CORBA 1. Introduction Recent advances in the computer network and the Internet, have given the Web increasing pop- ularity, but the growth in Web sites has made searching the Internet a more involved task, with the result that search engines now claim more importance as tools, and their number increases accordingly. Standard search engines, for example, Yahoo, 1 AltaVista, 2 Lycos, 3 InfoSeek, 4 Galaxy, 5 and WebCrawler, 6 may help users find what they Computer Networks 34 (2000) 467–480 www.elsevier.com/locate/comnet * Corresponding author. Tel.: +886-3-557-2930; fax: +886-3- 559-1402. E-mail address: [email protected] (Y.-S. Chang). 1 http://www.yahoo.com. 2 http://www.altavista.digital.com. 3 http://www.lycos.com. 4 http://www.infoseek.com. 5 http://galaxy.einet.net/galaxy.html. 6 http://webcrawler.com. 1389-1286/00/$ - see front matter Ó 2000 Elsevier Science B.V. All rights reserved. PII: S 1 3 8 9 - 1 2 8 6 ( 0 0 ) 0 0 1 3 1 - 6
Transcript
A new multi-search engine for querying data through anInternet search service on CORBA
Yue-Shan Chang a,*, Shyan-Ming Yuan a, Winston Lo b
a Department of Computer and Information Science, National Chiao Tung University, 1001 TA Hsueh Road, Hsin-Chu 30050,
Taiwan, ROCb Department of Computer and Information Science, Tung Hai University, Taichung, Taiwan, ROC
Received 14 February 2000; received in revised form 5 May 2000; accepted 19 May 2000
Abstract
Search engines are important but generally far from ideal tools of the World Wide Web (WWW). Many researchers
therefore prefer to use meta-brokers to construct multi-search engines (MSE). However, these have no uniform pro-
gramming interfaces, which makes tying them with other search engines di�cult. Moreover, for an application that
needs a search service capability, querying them is di�cult.
To reduce that di�culty, we propose in this paper an Internet search service (ISS) based on common object request
broker architecture (CORBA) that follows the style of common object service speci®cation (COSS). We design a multi-
search engine based on ISS, which we term Octopus. For a system developer, because of its uniformity of interface,
Octopus easily ties with any search engine. Equally, for an application programmer, the ISS o�ers a clear interface for
application programs to search for information or mine data from the Internet. We demonstrate our approach to
designing multi-search engines through ISS by tying two search engine agents, Yahoo and AltaVista, with Octopus and
show how CORBA clients query them. Programmers may use this interface to construct their search engine agents or
query a search engine agent in their applications. Ó 2000 Elsevier Science B.V. All rights reserved.
Keywords: Internet; Multi-search service; Search engine; World Wide Web; CORBA
1. Introduction
Recent advances in the computer network andthe Internet, have given the Web increasing pop-ularity, but the growth in Web sites has madesearching the Internet a more involved task, withthe result that search engines now claim more
importance as tools, and their number increasesaccordingly.
Standard search engines, for example, Yahoo, 1
AltaVista, 2 Lycos, 3 InfoSeek, 4 Galaxy, 5 andWebCrawler, 6 may help users ®nd what they
1389-1286/00/$ - see front matter Ó 2000 Elsevier Science B.V. All rights reserved.
PII: S 1 3 8 9 - 1 2 8 6 ( 0 0 ) 0 0 1 3 1 - 6
want, but they have their limitations. For example,none of them individually is su�cient and most ofthem return too many irrelevant, outdated, orunavailable references [1]. In general, users have toquery many engines before obtaining the mostrelevant matches. In addition, each search enginehas its own interface. A novice is confused by thisvariety of search engines. Designing from scratch acomprehensive search engine with a search capa-bility equal to other search engines combined isnot easy. Many researchers thus construct MSEsthat use meta-brokers, such as MetaCrawler [1],Customizable Multi-Engine Search Tool [2], Sav-vySearch [3,4], Softbot [5], Amathaea [6], and thesolution proposed by Overmeer [7,8]. These toolstie a number of search engines together in a systemand act as a dispatcher. When a user initiates asearch, a MSE dispatches the request to varioussearch engines and collects the results.
1.1. Motivations and objectives
Even with MSEs, however, there are problems,especially with the scalability and ¯exibility of asystem. Even if an MSE is capable of tying withany existing search engine, it may be necessary inthe future to integrate it with new and morepowerful search engines. Most MSEs lack exten-sibility, because they have no uniform interface forstandard search engine agents. Generally, there areproblems with tying an MSE to a new searchengine.
Although Internet users may ®nd MSEs useful,this is less true for programmers, who need to havea search component included in their applications.When an application needs to conduct a search,the programmer must either design a search com-ponent in their applications or query existingsearch engines. Regardless of which approach isused, the programmer will always need to extractwhat is wanted from the complicated HTML ®lethat is returned. For example, a programmer de-veloping an application that has a search capa-bility must explore and analyze the interface of thesearch engine. Then, the query string must be en-capsulated into URL format in terms of the queryinterface of the search engine and be sent to thesearch engine in HTTP protocol. When the results
are returned, the program must also extract theinformation from the complicated HTML ®le.Each operation needs to handle the network con-nection. These are tedious tasks. An MSE is thusdi�cult to use when a speci®c application mustsearch.
Among programming techniques, object orien-tation [9] o�ers greater portability and reusabilityand has been widely applied to software designand development. Many new applications are de-signed in object-oriented language, such as C++ orJava. In recent years, a range of distributed objectmiddle-ware has become available, for example,the Object Management GroupÕs CORBA [10],MicrosoftÕs DCOM [11] and SunÕs JAVA RMI.CORBA is an industrial standard and has morethan 1000 members worldwide. Moreover, it hasbeen ported to many operating environments, suchas MicrosoftÕs Windows, UNIX and MVS.
Here, we propose an ISS on CORBA, which isan industrial standard of a distributed object-ori-ented platform [10], and design a multi-searchengine based on ISS. The design of the ISS inter-face follows the style of COSS [12].
Several reasons motivated this work. First,every search service on the Internet has its owninterface, which is confusing to novices. Wetherefore propose a uniform interface that canaccommodate most of the interfaces found insearch engines. Second, if an application requiresto conduct a search, then incorporating searchservices must be made easy. By means of ISS,programmers can use this interface to constructsearch components or to query the search enginesin their applications. Based on the CORBA stan-dard, applications can be developed in other en-vironments, such as CORBA, COM, and Java, asshown in Fig 1, but those so developed must bemediated through IIOP bridges [10,13]. Third,since the interface is uniform, designing a multi-search engine based on ISS is simple. Fourth, wehope to establish a standard for Internet SearchService on the OMGÕs COSS. Finally, we believethat ISS can be easily applied to other types ofsearch engines, such as knowledge-discovery sys-tems, real-estate systems, and digital libraries, andto heterogeneous search engine agents. Thus, formost search or query services, whether or not on
the Internet, this approach can be extended tomulti-search engines, the design for which can in-herit the advantages described above.
The major objective of this paper is to proposea uniform interface for Internet search services,which will o�er a programming interface for lo-cating of what it wishes to retrieve. Since the ar-chitecture of ISS is of a 3-tier client/server modeland its interface is uniform, any speci®c applica-tion that needs to search for information or tomine data from the Internet need only initiate thesearch operation of an agent via ISS. Programmersdo not need either to explore the interfaces ofvarious search engines, or to construct searchcomponents in their applications. In Section 2, anISS interface and a programming example of ISSare described. In Section 3, we build our experi-mental multi-search engine, which we term Octo-pus, and demonstrate the proposed approach todesigning a multi-search engine using ISS.
We begin by exploring the interfaces and attri-butes of existing search engines and then combinethese interfaces into a uniform interface to form anISS. Then, using the ISS interface, we constructtwo representative search engine agents ± one forYahoo and the other for AltaVista. As the ISS isbased on CORBA, and the agents are built asInternet search components, programmers can use
the interface to search Web sites in their applica-tions. In addition, we tie the agents together andbuild Octopus. It is built as a multi-threaded and amulti-agent version to serve incoming requests,then to collate and ®lter results before returningthem to users. Although the agents were con-structed as CORBA objects, and the interface wasuniform, other search engines could be easily tiedto Octopus.
1.2. Search engine overview
Search engines are powerful tools for assistingusers to navigate the rapidly expending WorldWide Web. Most large-scale search engines can bedivided into two categories: directory scheme, suchas Yahoo and Yam,7 and active search scheme,such as AltaVista and InfoSeek, etc. With direc-tory scheme, the Web manager must register theWebÕs address, description, and other identifyinginformation, while active search schemes searchInternet Web sites periodically and index relateditems of information in the database.
In directory schemes, registered Web sites arecategorized manually. Querying this kind of search
engine may produce more relevant results. Sincedirectory schemes can only accommodate regis-tered sites, and many sites are unregistered, resultsfrom them may be fewer than those obtained fromactive search schemes. In comparison, activesearch schemes search most Web sites periodicallyand so return more results. Nevertheless, they donot guarantee relevance, even if the results have ahigher index. However, combining the advantagesof both search schemes could improve searchperformance and results.
Most search engines have their own interfaces,which may not be alike. We investigate those in-terfaces to de®ne our ISS interface. Two repre-sentatives search engines are explored ± Yahooand AltaVista ± and the results are shown inTable 1. A full description of the interfaces of bothengines can be found on their home page.
There are two types of interfaces: common andspecialized. Common interfaces contain many at-tributes that are included in most search engines;specialized interfaces do not. These two types areshown in Table 1, and we believe that they includemost existing search engine interfaces. Neverthe-less, we have not covered all search engines so far,but we shall add to these interfaces, and try toinclude as many engines as possible. The design ofour ISS is described in Section 2.
The paper is organized as follows. Section 2presents the design of our ISS based on CORBAand a sample program to demonstrate how to useit. Section 3 describes an experimental multi-search engine based on our architecture, which weterm Octopus. Section 4 presents a discussion.Finally, Section 5 gives conclusions.
2. Designing an Internet search service
In devising a multi-search engine, we ®rst de®nean ISS and describe its design in detail beforepresenting an example to demonstrate its use.
2.1. Architecture of ISS
We de®ne an ISS interface by following thestyle of COSS. One of major objective of our ISS isto provide a uniform interface for search engines.
Programmers can use this interface to constructsearch engine agents or to query search service intheir applications based on our object implemen-tation. We design the ISS based on the descriptionof the interface comparison given in Section 1.2.The design consists of three components: Search-Factory, Search, and ResultCollection. The Searchis at the core of executing a search in ISS. TheSearchFactory component creates Search. TheResultCollection component collects results. Fig. 2shows the relationship of these components. Anarrow with a vertical bar is used to show that thetarget object supports the interface named next tothe arrow and that clients holding an object ref-erence of this type can perform operations de®ned
by the interface. Our design includes as manysearch interfaces as possible. Next, we describe theISS interface and its processing scenario.
2.2. The ISS interface
The SearchFactory creates a Search object. Be-fore a client program obtains a Search object, itmust bind to the SearchFactory to obtain its objectreference. This interface is shown in Table 2.
After the client program obtains the referenceof the SearchFactory object, it then invokes New-Search( ) operation to obtain the Search object,which is the component to invokes which searchengine agents. In ISS, the Search interface have®ve methods, which are AddKeyword( ), Remove-Keyword( ), GetKeyword( ), ExecuteSearch( ),and AbortSearch( ). These operate as follows.
When a client program obtains the reference ofthe Search object, it then may perform AddKey-word( ) operation to put query string into theSearch object. The query string is removed by theRemoveKeyword( ) operation. These two methodshave an input keyword parameter in respect of theadded or the removed string. Each keyword con-sists of a query string and an attribute. The at-tribute denotes whether the string is included ornot.
In addition, a client may look for the querystring by the GetKeyword( ) operation, with aninput parameter that is the needed key word index.Before invoking a search engine agent, a clientmust put one or more attributes related to thequery into the Search object. Then, the clientprogram initiates the ExecuteSearch( ) operationto perform the search. To discontinue a search, theclient program can perform the AbortSearch( )operation. These methods are shown in Table 3.
To accommodate all search engine interfaces,we refer to Table 1 and de®ne a few attributes inthe Search interface. These attributes are Domain,Tag, Date, Near, DispNum, Area, DataBase, and aread-only No_Keyword. Most of them are bit-wiserepresentations. For instance, a Tag attribute canrepresent the search target which may be in theform of title, URL, anchor, applet, image, link, ortext. We declare a long integer to represent thisattribute. Each bit represents a tag, and the re-mainder bits are reserved for future use. InCORBA, attributes can be translated into twooperations by an IDL compiler. They are get_xxxand set_xxx, respectively. Here, xxx is the attrib-uteÕs name. How to use these attributes is imple-mentation-dependent.
The third ISS interface is the ResultCollection,which is shown in Table 4. In this interface, theretrieve_element_at( ) is the only method function.When a client program performs the Execute-Search( ) operation, the Search object issues aquery request to related search engines, createsand returns the ResultCollection object reference tothe client program. The client program reads thenumber of results from the read-only attributenamed No_Result, and then retrieves the returnedreferences that include a data structure consistingof Title, URL, Description, Date, and Weight,
using retrieve_element_at( ) method. In the fol-lowing two subsections, we explain the scenario ofISS and present a programming example.
2.3. Normal scenario of ISS
Fig. 3 shows the normal scenario for ISS. Forthe client program, it ®rst binds to the Search-Factory to get its object reference, and execute theNewSearch( ) operation to construct a new Search
object. When the SearchFactory receives a New-Search request, it creates a Search object and re-turns object reference to the client. Once the clienthas this, it can execute other operations, such asAddKeyword, RemoveKeyword, and GetKeyword,and set related attributes into the Search object.Then, the client program uses the ExecuteSearchmethod to issue a search request. Once the Searchobject receives a request, it encapsulates relatedkeywords and attributes into the search enginequery string and sends it to corresponding search
Fig. 3. Scenario of ISS architecture.
Table 3
The Search interface
struct Keyword //keyword structure
{
char Inclu_Exclu;
string item;
};
interface Search //search interface
{
attribute string Domain; //domain and host
attribute long Tag; //title, URL, anchor, applet, image, link, text
attribute string Date;
attribute long Near; //how many words between two string
attribute unsigned long DispNum; //how many references displayed in a page
attribute char Area; // Web site, Categories
attribute boolean DataBase;
readonly attribute long No_Keyword; //get the number of added keyword
boolean AddKeyword(in Keyword add); //add keyword for future search
engines. When the Search object receives thesearch result from the search engine, it de-encap-sulates returned messages and puts them into Re-sultCollection object. Then, the Search objectreturns ResultCollection object reference to theclient program, which then extracts query results.
2.4. Programming example
In this subsection, ®rst we demonstrate thecodes for a client program and show how to querysearch engine agents. According to the described inSection 2.2 interface, we ®rst compile the interfacede®nition language (IDL) ®le and generate theclient stub and server skeleton. Then, we imple-ment the search engine agent component andcompile it with the server skeleton. Finally, wewrite a server program and register it into ORB.The server program is shown in Table 5.
As described in Section 1, an application pro-gram using this interface can easily search for in-formation from the Internet. This functionality isnot supported by other MSEs. We now demon-strate the implementation of a client program. Thecode sequences shown in Table 6 are the same asthe scenario for ISS.
First, the client must bind with a search engineagent and obtain a SearchFactory object reference.If the system supports multiple agents, then theclient may also bind with other agents and set theattributes of agent. Second, the NewSearch( )operation is invoked to obtain the Search objectreference. Third, the client puts the query stringinto the Search object by invoking the AddKey-word( ) operation. Fourth, all the attributes are
set. Fifth, the search is executed by invoking theExecuteSearch( ) operation. Finally, the result isobtained from the ResultCollection object.
By this procedure, it is clear that an applicationwanting to search the Internet neither needs tohave a complex search component nor to executemany sophisticated network-accessing e�orts. Itneeds only to issue a few of invocations on theagent.
3. Application of ISS
We build our experimental heterogeneoussearch engine, i.e. Octopus, to demonstrate thefeasibility of ISS. Our system involves two searchengine agents, which are implemented as a COR-BA object, but using a similar method could allowus to add other search engine agents to the systemeasily.
3.1. System architecture
Since Octopus utilizes ISS IDL de®nitions, itprovides a single and uniform interface forsearching Web documents. On receiving a queryfrom a WWW user, Octopus dispatches it tomultiple search engines in parallel, and collates thereturned references.
Fig. 4 shows the Octopus system architecture.In this system, a Web user posts a query requestvia common gateway interface (CGI). The CGIthen forks a mediator for each request. The re-sponsibilities of the mediator are as follows: First,it obtains agent information from the Service Re-pository that stores agent information, such as the
number of agents and its name. Then it createsmultiple threads to perform this query. In ourdesign, each created thread is a wrapper. Second, itcollates the returned information from all of theagents, before merging and ®ltering them. Finally,it returns the query results to the Web user.
In addition, each wrapper is a CORBA clientprogram, which is responsible for setting the at-tributes for each search engine agent and issuinginvocations for operations. Most search engineshave their own attributes, although they di�erfrom each other only slightly. Each agent has onewrapper. The ®rst task of a wrapper is to addkeywords into the related agent, and set its attri-butes. Next, it performs the ExecuteSearch( )operation to invoke a search. Once a search enginereturns the results to the agent, the wrapper willextract the returned information from the agent,and return it to the mediator. A further clear ad-vantage to our system is that programmers canimplement their own wrappers in their applica-tions to perform Web search functions and thushave a search service capability.
Another component in our system is the agent,which is also a CORBA object. When the agentreceives the request from a wrapper, it will en-capsulate query information into the HTTP for-mat of the related search engine. Finally, it sendsthe query information to the related agent andobtains the results from the search engine agent.Although we have a uniform interface- ISS, addingother agents into our system is made easy.
Moreover, because these agents are imple-mented as CORBA objects, a general CORBAclient can also use this interface to invoke searchengine agents in their application programs. Ofcourse, clients that are developed in other objectmodels, such as MicrosoftÕs COM/DCOM, canalso use these agents via CORBAÕs Internet Inter-ORB Protocol [10] in the same way.
3.2. System implementation
With Octopus, the mediator is an importantcomponent. As mentioned above, Octopus serveseach query request by a dedicated thread of amediator. The mediator dispatches the request tomultiple search engine agents. Thus the mediatorFig. 4. Octopus system architecture.
has to create multiple client threads. Each thread isa client of agent. The codes for creating multiplethreads are shown in Table 7.
On the other hand, although no single searchservice is su�cient in the WWW, and a heteroge-neous multi-search service may include most ref-erences, many search engines may returnduplicated references, and two search engineagents may return the same references, which re-sults in confusing users. In Octopus, duplication isavoided by using a hashing function in the medi-ator for ®ltering the references. Any returned ref-erence with the same network address as aprevious one is discarded to guarantee its unique-ness. In our experiment, the returned referencesfrom Yahoo and AltaVista search engines are 2286and 4000, respectively for 20 query items, whileOctopus has 5183. Clearly, the two search enginesduplicated 1103 references, which gives a duplica-tion rate 21%. In addition, Octopus uses a dedi-cated thread to guarantee the availability of the®ltered references.
Another important component of the architec-ture shown in Fig. 4 is the wrapper. This acts asthe client of the search engine agent. The imple-mentation is similar to the CORBAÕs client de-scribed in 2.4.
Another feature that is provided in many searchengines is the weight of responses. The weightrepresents the relevance of the responses to thequery string. Each search engine has a proprietaryweighting algorithms. In most search engines,query results are shown on the result pages in or-der of weighting. However, Yahoo and AltaVistaresponses do not show weights, and so in a MSE itis hard to accurately evaluate the weight of eachitem. If the search engines do not respond with theweight in a result, the weighting algorithm in Oc-topus simply gives each item a weight of 500. InOctopus, the weight is implemented by normaliz-
ing the scores returned by search engines to be-tween 0 and 1000. Then the mediator calculates theaverage for all the weights from the search engineagents. We do not attempt to improve the inte-gration beyond this ad hoc approach, because ourfocus was on proposing modularized architectureand interfaces. We ensure with Octopus that itemsare shown in order of weighting. It is a simpleapproach that can be easily changed in the futurein modularized architecture. We can also applyother weighting algorithms to Octopus, such as[14].
How to merge search results from search en-gines is also an important issue in a multi-searchengine. Raw results from individual search engineagents must be integrated for display to the user.Results can be displayed with little additionalformatting and can be rank ordered or interleaved.In Octopus, the number of items received on onepage is 100 from Yahoo and 200 from AltaVista.This is in order to promote the system perfor-mance. The maximum number of retrievable itemsfrom the two search engines is 200. In Octopus, theagent only fetches the ®rst page (HTML ®le) thatcovers the URL and the description of each item.Therefore, the approach to merging all the searchresults is to ®lter those that come from the agentsand weigh the ®ltered results. Similarly, the queryresults are shown in order of weighting.
The mediator has a central role in Octopus.Other valued-add services may also be integratedinto the mediator to enhance the systemÕs capa-bility.
3.3. User interface
The underlying systems in Octopus are twoplatforms (a SPARC and a Windows NT). TheORB of this system is IONAÕs Orbix 2.02 [15],which fully complies with CORBA speci®cation.The conventional search engines here included inOctopus are two typical search engines ± Yahooand AltaVista search engine. Figs. 5 and 6 showthe user interface. When a user submits a queryusing the query form (Fig. 5), the system willperform the query process and obtain the resultsfrom Octopus. The returned results are organizedand displayed in a uni®ed form, as shown in Fig. 6.
The user may either issue a simple or a complexsearch. For a complex search, all of attributesshown in Fig. 5 are adjustable and for a simplesearch they have a default value.
3.4. Performance evaluation
Performance is very important issue in the cli-ent/server model. In Octopus, there is one media-tor, multiple wrappers, one ORB, and multipleagents working in parallel in maximize the service.Performance is therefore seriously e�ected. Toimprove performance in our system, the followingstrategies are used: multi-threading and variousagents con®gured on various hosts.
Multi-threaded programming is a well-knowntechnology for improving server performance. In
Octopus, the mediator, wrappers and search en-gine agents are all multi-threaded versions. Whena mediator thread is created, it immediately createsmulti-threaded wrapper. Each thread of wrapper isassociated with one agent that is also a multi-threaded version created by ORB.
In addition, we assign each agent to a dedicatedhost. This is easily done in the CORBA environ-ment. Thus, the strategy can balance the overheadof the system. We are also aware of object mi-gration [16] techniques to balance the system loadfor future consideration.
We perform the preliminary measurementsshown in Fig. 7 to assess the performance ofOctopus and compare it with the Yahoo andAltaVista search engines. Though the overhead oftransmission in Internet in most situations is
unpredictable, it is obvious from the ®gure thatOctopus is e�cient. The average query time forOctopus is slower than the two representative en-gines. The reasons for overhead in Octopus are tocreate multiple threads that execute search opera-tions, to deliver message on ORB, and to ®lterreturned references. As Fig. 7 shows the resultingperformance is reasonable. We also measure thetotal overhead, which as in Fig. 8 shown, is 6.5%.We believe that the major overheads are ®lteringthe returned references and the networking over-head of CORBA. But these operations are exe-cuted in parallel, so the overhead does not increaselinearly, i.e., it does not rapidly increase with thesearch engines increase.
Fig. 9 shows a comparison of averaged per-formance between Octopus and two well-known
multi-search engines, MetaCrawler and Savvy-Search, respectively. As the returned records ofeach query request from the SavvySearch areabout 70 and each return has a constant num-ber-15, the comparison is limited to a maximum60. From the results shown in Fig. 9, it is ob-vious that Octopus is e�cient though it hasabout 0.5 s network overhead that is measuredfrom the test-bed to two real systems. A possiblereason for this result is that Octopus is only aprototype multi-search engine based on ISS. Ithas no applied sophisticated algorithms andmany system access operations to handle theinformation returned from the search engineagents. In addition, evaluating network behavior,such as the numbers of packet retransmission, isdi�cult.
CORBA is an industrial standard, which sup-ports more than a dozen services in the COSS.There are many bene®ts to making a large client/server middle-ware based on CORBA [20]. In de-signing and implementing our ISS-based Octopus,we can mention further advantages.
First, with the progress in search engine tech-nology, more powerful search engines can also betied into the system in a similar way. The systemdeveloper can easily tie new search engines into thesystem simply by creating search engine agents. Inour experience, 80% of all codes found in agentsare the same, because the agents have the sameserver stub that is generated by IDL compiler. Tocreate a new agent, only a small part of a programneeds rewriting. All the network operations arehidden from the CORBAÕs server stub. The de-veloper needs only to handle the interface of thesearch engine when constructing the new agent.The remainder of the system can be retained.
Second, it is easy for programmers to buildapplications that need a search ability. Applicationprogrammers utilizing the interface to search forinformation in their application can hide thecomplexity from network programming and con-centrate most e�ort on other signi®cant value-added services. After the search engine returns theresults, the program does not need to extract theinformation from the complicated HTML ®le.Since, they are all based on the same interface,applications are undiscerning when queryingagents. In addition, through the CORBA stan-dard, applications can also be developed in otherenvironments, such as COM and JAVA. But, ap-plications so developed must be mediated throughIIOP bridges.
Third, CORBA is a distributed object-orientedenvironment. In Octopus, agents can be easilydistributed at di�erent locations. In this way,balancing the load while the size of system is onthe increase is easy. The system manager candynamically add other agents into the system.Thus, a system based on ISS naturally has sca-lability.
Finally, because this is a modularized andcomponent-based approach, it will be easy in thefuture to replace certain components with new anduseful algorithms, such as a weighting algorithmand a natural language processing algorithm.Similarly, a system based on ISS naturally has¯exibility.
4.2. Extension and future works
In addition, as described in Section 1, the ISS iseasily applied to other types of search engines. Forexample, many libraries allow user to inquire thebook information on the WWW [17,18]. Such aservice provides a query on a Z39.50 server to bemade via a Z39.50 gateway [19] that translates thequery string into Z39.50 format. The scenario isthe same as for a general search engine. Thus, in-tegrating such a service into the system can followa similar procedure.
Users generally query book information fromlibraries via Subject, Title (book or journal), Au-thor/s, ISBN, ISSN, or Keywords from libraries.These styles can be seen as a query attributes.Therefore, the Tag attribute in the Search Inter-face of ISS can merge all the attributes because it isdeclared as Long type and the representation is bit-wise. All ISS interfaces do not need modi®cation.Before constructing an agent we need only analyzethe query string that is sent to the server. Theimplementation steps and approaches to the agentare same as for the general search engine agent inOctopus. Obviously, this approach can be ex-tended to merge heterogeneous search services.
Octopus is only a prototype multi-search enginethat ties with two agents. To increase its usability,we shall construct more search engine agents in thefuture. We shall also add a few value-added ser-vices, such as personalized function that is a pop-ular feature in most search engines.
5. Conclusions
In this paper, we have proposed an Internetsearch service (ISS) based on the CORBA, which isan industrial standard of a distributed object-ori-ented platform and has been announced by OMG.
We have followed the style of COSS to de®ne theinterface of the ISS. In addition, according to ISSinterface, we have constructed two representativesof search engine agents ± Yahoo agent and Alta-Vista agent. Since, the ISS is based on the CORBAand was implemented as an Internet search com-ponent, programmers can use the interface tosearch Web site on the Internet in their applicationsvia these two agents. We have integrated these twoagents and built a multi-search engine prototype ±Octopus. As these agents are implemented as aCORBA object, and the interface is uniform, othersearch engines are readily tied into Octopus.
The major contributions of our work have beenas follows: First, we proposed a uniform interfacethat accommodates most search engine interfaces.The function of ISS is useful in the general appli-cations that need a search service capability. Pro-grammers can use this interface to construct searchengine agents or to query search engines on theInternet in their applications. Second, because theinterface is uniform, we can easily build a multi-search engine. In addition, we can easily tie a newsearch engine into the multi-search engine. Existingmulti-search engines do not have this capability.
In our experience, using ISS to implement eithersearch engine agents or multi-search engines iseasy. We also believe that ISS can be easily ex-tended to other types of search engine agents, suchas knowledge- or data-discovery, real-estate sys-tems and digital libraries.
Acknowledgements
The authors thank the referees for their valu-able comments. This work was partially supportedby National Science Council of Taiwan ROC un-der grant NSC88-2213-E-009-087 and by Instituteof Information Industrial of Taiwan ROC undergrant No. C87-144.
References
[1] E. Selberg, O. Etzioni, Multi-engines search and compar-
ison using the metacrawler, in: Proceedings of the Fourth
[20] R. Orfali, D. Harkey, Client/Server Programming with
JAVA and CORBA, Wiley, New York, 1997.
Chang Yue-Shan was born on August4, 1965 in Tainan, Taiwan, Republic ofChina. He received the B.S. degree inElectronic Technology from NationalTaiwan Institute of Technology in1990 and the M.S. degree in ElectricalEngineering from the National ChengKung University in 1992. Currently, heis a candidate of Ph.D. in Computerand Information Science at NationalChiao Tung University. His researchinterests are in Distributed Systems,Object Oriented Programming, FaultTolerant, and Internet Technologies.
Shyan-Ming Yuan was born on July11, 1959 in Mauli, Taiwan, Republicof China. He received the B.S.E.E de-gree from National Taiwan Universityin 1981, the M.S. degree in ComputerScience from University of Maryland,Baltimore County in 1985, and thePh.D. degree in Computer Sciencefrom University of Maryland, CollegePark in 1989. Dr. Yuan joined theElectronics Research and Service Or-ganization, Industrial Technology Re-search Institute as a Research Memberin October 1989. Since September
1990, he had been an Associate Professor at the Department ofComputer and Information Science, National Chiao TungUniversity, Hsinchu, Taiwan. He became a Professor in June,1995. His current research interests include Distributed Objects,Internet Technologies, and Software System Integration. Dr.Yuan is a member of ACM and IEEE.
Win-tsung Lo received the BS and MSdegrees in applied mathematics fromNational Tsing Hua University, Tai-wan, Republic of China, and MS andPh.D. degree in computer science fromthe University of Maryland. He is nowan associate professor of computerscience and the director of ComputerCenter at Tung Hai University, Tai-wan, Republic of China. His researchinterests include architecture of dis-tributed systems, data exchange inheterogeneous environments, andmulticast routing in computer net-works.
