Help parsing continuous text from the html parsed page.

[Martin Ho]

I've got this problem, where I need to extract an articles from many
different news sources (webpages).
I need to write some logic which would know how to extract the text
only and not a garbage around.

I am able to parse the text from html and all I get now is the text
from the site. But there is lot of short text, lot of advertising
text and garbage all around. In the middle is the article I need. I
was wondering on writing some program which would be smart enough to
figure out where article starts and ends... but apparently it's not
as easy as I thought. I could somehow depending on spaces and dots
make some fairly workable way of doing it. But I am not even sure how
I would go about that.

So when you look at this text in quotes, you'll see the article in the
middle. I will get these articles from many different sources so
wording will be always different, but it would generaly look
something like this:

The Globe and Mail
@import url('/cssv3/net5upcss.css');
if (navigator.userAgent.indexOf('MSIE 5') != -1) {
document.write(" ")
}

if((navigator.userAgent.indexOf('MSIE') != -1) &&
(navigator.userAgent.indexOf('Mac') != -1)) {
document.write(" ")
}


News
Investing
Technology
Vehicles
Careers
Home |
Business |
International |
Sports |
Travel |
Entertainment
d.body.style.visibility="hidden";
Search Site

Search Tips
showTextAd("USERNAME");







A strike at Bell Canada has been at least temporarily averted after

the country's largest phone company made a new offer to 7,000
technicians that will be voted on over the next several weeks.

Joel Carr of the Communications Energy and Paperworkers Union of

Canada, which represents the technicians, said the offer is
significant enough under CEP rules to warrant a vote by workers.
While he didn't discuss details, he said in general the offer is
“very much improved” from Bell's June offer,
which was rejected by 62 per cent of those that cast ballots.

The CEP filed notice of strike on Monday, meaning workers could have

been on the pickets Thursday afternoon. Bell made what it called its
final offer late yesterday, and local union leaders from across
Ontario and Quebec meet tomorrow morning in Toronto to issue a
recommendation to workers. Pensions are the No. 1 issue.

“It represents the limit to which the company can go in

meeting the priorities of the union and its members,” Ellen
Malcolmson, senior vice-president of operations at Bell, said in a
statement.

Voting, Mr. Carr said, will begin as soon as possible and take about two weeks.

Technicians represent about one sixth of Bell's work force of

42,500. The workers went on strike for five weeks in 1999, fighting
for job security, and the resulting contract expired last November.
There can be no strike while the workers vote on the new offer.

Bell is owned by BCE Inc. of Montreal, which has a majority stake in

Bell Globemedia, owner of The Globe and Mail and CTV television.
Shares of BCE were up 25 cents or 1 per cent to $26.36 at midday on
the Toronto Stock Exchange.

While the Bell negotiations could reach a resolution, a strike at

Aliant Telecom Inc. is nearly 13 weeks old. Aliant, majority owned by
Bell, is the dominant phone company in Atlantic Canada and 4,300 of
its workers — two-thirds of the work force — have
been on the picket lines since April 23.

Union leaders and the company met last week for the first time since

the strike began, but no progress was made and no other meetings are
planned.

if (!mt_8734802b) {if (parent.mt_8734802b) {var mt_8734802b =

parent.mt_8734802b;} else {var mt_8734802b=false;}}

if (mt_8734802b) {
document.write('');
}



if (!mt_8734802c) {if (parent.mt_8734802c) {var mt_8734802c =

parent.mt_8734802c;} else {var mt_8734802c=false;}}

if (mt_8734802c) {
document.write('');
}

All I need is the way to figure out that article starts at from "A
strike at Bell Canada has been " and ends at "and no other meetings
are planned.", and that way it would be easy to extract important
text, but how to write this logic???
Uff...

Does anyone have any idea or can give me some help on how to write
some smart text extractor from the page?

Thanks a lot.

Martin Ho

Copy of this problem at:
http://www.groupsrv.com/dotnet/viewtopic.php?t=57362

 

[Krakatioison]

Wow, interesting. I would probably try to somehow count the spaces and if
there is no space every at least for 20 characters then get rid of them.
That way it could eventually somewhat work.
But this will need someone smarter then I to answer this question.
K.

 

[Cor Ligthert]

Hi Martin,

Forget this one, this is not HTML this is JavaScript document Write which
uses some HTML, it is almost impossible to find a way to extract those text.

Large search companies have problems with finding things in this kind of
pages, so that you find a way when you are not extremely well in all aspects
of JavaScript and HTML is possible however not so sure.

I hope this helps anyway?

Cor