DFN Data Sniffer

Dileepa · 1 DFN Data Sniffer Sat Nov 03, 2012 12:48 pm

Dileepa

Senior Manager - Equity Analytics

Posts : 346
Equity Stars : 6273
Reputation : 44
Join date : 2011-03-21
Location : Tokyo

This is a tool I developed some time ago.
This will fetch 10+ years of past data from DFN even if you are offline.
I wanted to access DFN db offline for simulation purposes, but DFN does not allow that.
This will be helpful, specially if you are going to stuck in an aircraft for 10-12 hours.

~~Unfortunately this will work only on "ASIA SECURITIES" for the time being.~~
Want to make it universal for all DFN versions, but first I need to make sure it works on different platforms.
~~ASIA SEC. users~~ please try & let me know if this works on your machine.
Just select the counter on cell "C4" & click "Fetch Data"

Edit :2012-11-03:6.30pm
Received some help Smile

Now all versions DFN should be supported scratch

New version is attached, let me know if there are any issues.
If attached file does not work. please try this.
http://www.mediafire.com/view/?dpi7j1nrrob9f9d

Last edited by Dileepa on Sun Nov 04, 2012 6:08 am; edited 1 time in total

Jana · 2 Re: DFN Data Sniffer Sat Nov 03, 2012 3:06 pm

Jana

Senior Vice President - Equity Analytics

Posts : 2640
Equity Stars : 10731
Reputation : 195
Join date : 2011-03-10

How can we trust this... !!??
and how can we make sure it can't be fetch our personal details as well its mean portfolio and account statement details ???

Dileepa · 3 Re: DFN Data Sniffer Sat Nov 03, 2012 3:22 pm

Dileepa

Senior Manager - Equity Analytics

Posts : 346
Equity Stars : 6273
Reputation : 44
Join date : 2011-03-21
Location : Tokyo

Jana wrote:How can we trust this... !!??
and how can we make sure it can't be fetch our personal details as well its mean portfolio and account statement details ???

I don't think statement details or personal details are stored in DFN.
It only hold user ID in local PC and authentication (data transfer) is via encrypted DTP as far as I know.
But yes, VBA scripts could run potentially harmful codes. Use at your own risk.

Again this is for offline use.
Artificial Neural Network, Signal Generator and some other features are disabled.

Slstock · 4 Re: DFN Data Sniffer Sat Nov 03, 2012 3:31 pm

Slstock

Director - Equity Analytics

Posts : 6610
Equity Stars : 13397
Reputation : 1158
Join date : 2011-01-22

Dileepa. Thansk.

Is there an API for Dfn . I presume your VB code access the Data server component? I beleive they have a separate authentication server.

Am curious to know ( if possible or via PM) the DFN connection APIs which you used .

This is for education purposes.

Dileepa · 5 Re: DFN Data Sniffer Sat Nov 03, 2012 3:47 pm

Dileepa

Senior Manager - Equity Analytics

Posts : 346
Equity Stars : 6273
Reputation : 44
Join date : 2011-03-21
Location : Tokyo

If above attachment does not work, try downloading from here.
http://www.mediafire.com/view/?dpi7j1nrrob9f9d
There seems to be some issues with upload limit.

Dileepa · 6 Re: DFN Data Sniffer Sat Nov 03, 2012 4:02 pm

Dileepa

Senior Manager - Equity Analytics

Posts : 346
Equity Stars : 6273
Reputation : 44
Join date : 2011-03-21
Location : Tokyo

slstock wrote:Dileepa. Thansk.

Is there an API for Dfn . I presume your VB code access the Data server component? I beleive they have a separate authentication server.

Am curious to know ( if possible or via PM) the DFN connection APIs which you used .

This is for education purposes.

slstock,
TWS API reference is here, but I believe DFN is slightly different.
http://www.interactivebrokers.com/php/apiUsersGuide/apiguide.htm#apiguide/apicover/apicover.htm
If you are interested about coding, DFN help on indicator builder & system tester will be very helpful.
From DFN Charting,
Help -> Index for general help &
Tools -> Indicator Builder -> Help for scripting help

Slstock · 7 Re: DFN Data Sniffer Sat Nov 03, 2012 4:31 pm

Slstock

Director - Equity Analytics

Posts : 6610
Equity Stars : 13397
Reputation : 1158
Join date : 2011-01-22

Oh, okay. Thanks. I presume you used the DDE for Excel to do this and Asia has TWS version?

Not sure DFN has such explict API info . But let me check.

Dileepa wrote:

slstock wrote:Dileepa. Thansk.

Is there an API for Dfn . I presume your VB code access the Data server component? I beleive they have a separate authentication server.

Am curious to know ( if possible or via PM) the DFN connection APIs which you used .

This is for education purposes.

slstock,
TWS API reference is here, but I believe DFN is slightly different.
http://www.interactivebrokers.com/php/apiUsersGuide/apiguide.htm#apiguide/apicover/apicover.htm
If you are interested about coding, DFN help on indicator builder & system tester will be very helpful.
From DFN Charting,
Help -> Index for general help &
Tools -> Indicator Builder -> Help for scripting help

Dileepa · 8 Re: DFN Data Sniffer Sat Nov 03, 2012 5:01 pm

Dileepa

Senior Manager - Equity Analytics

Posts : 346
Equity Stars : 6273
Reputation : 44
Join date : 2011-03-21
Location : Tokyo

slstock wrote:
Oh, okay. Thanks. I presume you used the DDE for Excel to do this and Asia has TWS version?

Not sure DFN has such explict API info . But let me check.

They don't have, that's the problem.
If they have our lives will be lot easier than this Sad

I'm not familiar with other trading systems, but I think they (AmiBroker, etc) are very advance than DFN.
In my opinion, a proper trading system should give greater flexibility for advance users.
My biggest problem is I can not execute a sell/buy order from a C code.
I had to use a separate GUI script (AutoIt) to execute an order(made a post about it some where in the forum).
Anyway SEC is against automatic trading systems. I hope their view have changed with the head change.

kav4 · 9 Re: DFN Data Sniffer Sat Nov 03, 2012 9:18 pm

kav4

Senior Manager - Equity Analytics

Posts : 369
Equity Stars : 5262
Reputation : 36
Join date : 2011-06-17

I get this error.

DFN Data Sniffer Untitl10

How to fix?

Dileepa · 10 Re: DFN Data Sniffer Sat Nov 03, 2012 9:23 pm

Dileepa

Senior Manager - Equity Analytics

Posts : 346
Equity Stars : 6273
Reputation : 44
Join date : 2011-03-21
Location : Tokyo

kav4 wrote:I get this error.

How to fix?

Please try this,
http://www.mediafire.com/view/?dpi7j1nrrob9f9d

smallville · 11 Re: DFN Data Sniffer Sun Nov 04, 2012 11:27 am

smallville

Associate Director - Equity Analytics

Posts : 4119
Equity Stars : 10129
Reputation : 530
Join date : 2011-02-03
Location : Back on Earth

Dileepa wrote:
I'm not familiar with other trading systems, but I think they (AmiBroker, etc) are very advance than DFN.
In my opinion, a proper trading system should give greater flexibility for advance users.

Absolutely, Ami is really powerful and could even be automated if the DDE source support it.. Samuel Daniel had done it and posted here where the intraday data could be used to get populated in Ami so that the data could be used for charting but unfortunately it didnt work in Windows 7. The Excel sheet used to hang while calling the DDE data source and I had no solution found for that ;(

Pls find the discussions here;

http://forum.srilankaequity.com/t11549-how-to-get-live-data-minute-by-minute-free-in-to-amibroker#77075

http://forum.srilankaequity.com/t12044-directfn-data-to-amibroker#80055

smallville · 12 Re: DFN Data Sniffer Sun Nov 04, 2012 11:37 am

smallville

Associate Director - Equity Analytics

Posts : 4119
Equity Stars : 10129
Reputation : 530
Join date : 2011-02-03
Location : Back on Earth

Dileepa wrote:If above attachment does not work, try downloading from here.
http://www.mediafire.com/view/?dpi7j1nrrob9f9d
There seems to be some issues with upload limit.

OK.. this one is working. .Thanks a lot mate..

But only the "fetch Data" button works.. Other buttons gave me the following error on Windows 7/Excel 2007.
Macros are enabled from "Trust Centre" section of the Excel too. Sad

[/url]

Need some help with the following;

1) In ASI data, Volume and TO has values only from 2009/10/02 onwards. Is it normal?

2) Need an excel formula or a work-around to convert a date in "2009/10/02" format to 091002 (YYMMDD) format..

3) Can you re-code this to fetch data of all counters at once? Also with the option of defining a specific period?

Dileepa · 13 Re: DFN Data Sniffer Sun Nov 04, 2012 2:47 pm

Dileepa

Senior Manager - Equity Analytics

Posts : 346
Equity Stars : 6273
Reputation : 44
Join date : 2011-03-21
Location : Tokyo

smallville,
Thanks for the feed back.

OK.. this one is working. .Thanks a lot mate..
But only the "fetch Data" button works.. Other buttons gave me the following error on Windows 7/Excel 2007.
Macros are enabled from "Trust Centre" section of the Excel too.
[/url]

Signal generator is still coding and ANN also need a lot of testing.
I had disabled those features for the moment. Will make those available as soon as possible.

1) In ASI data, Volume and TO has values only from 2009/10/02 onwards. Is it normal?

I also checked. Yes that data is available only for the period of 2009/10/02~2012/3/20.
Also No. of Trade data is available only after 2009/10/02~
CSE data feed is not consistent I believe.

2) Need an excel formula or a work-around to convert a date in "2009/10/02" format to 091002 (YYMMDD) format..
3) Can you re-code this to fetch data of all counters at once? Also with the option of defining a specific period?

Format issue should be a simple solve. 3) is possible but I think it will increase the load on processor & RAM.
Give me some time and I will give it a try.

Dileepa · 14 Re: DFN Data Sniffer Sun Nov 04, 2012 3:06 pm

Dileepa

Senior Manager - Equity Analytics

Posts : 346
Equity Stars : 6273
Reputation : 44
Join date : 2011-03-21
Location : Tokyo

smallville wrote:

Dileepa wrote:
I'm not familiar with other trading systems, but I think they (AmiBroker, etc) are very advance than DFN.
In my opinion, a proper trading system should give greater flexibility for advance users.

Absolutely, Ami is really powerful and could even be automated if the DDE source support it.. Samuel Daniel had done it and posted here where the intraday data could be used to get populated in Ami so that the data could be used for charting but unfortunately it didnt work in Windows 7. The Excel sheet used to hang while calling the DDE data source and I had no solution found for that ;(

Pls find the discussions here;

http://forum.srilankaequity.com/t11549-how-to-get-live-data-minute-by-minute-free-in-to-amibroker#77075

http://forum.srilankaequity.com/t12044-directfn-data-to-amibroker#80055

I think the issue is, you need to feed market time & sales data in real time to Ami.
That should not be very hard to do. DFN stores this data in it's DB in real time.
All that need is read that data (may be once in a min) and create a custom excel file.
Need to try the theory while market is open.
I have a office leave on 12th Nov. Will give it a try Smile

Dileepa · 15 Re: DFN Data Sniffer Sun Nov 04, 2012 5:51 pm

Dileepa

Senior Manager - Equity Analytics

Posts : 346
Equity Stars : 6273
Reputation : 44
Join date : 2011-03-21
Location : Tokyo

Dileepa wrote:

2) Need an excel formula or a work-around to convert a date in "2009/10/02" format to 091002 (YYMMDD) format..
3) Can you re-code this to fetch data of all counters at once? Also with the option of defining a specific period?

Format issue should be a simple solve. 3) is possible but I think it will increase the load on processor & RAM.
Give me some time and I will give it a try.

Give this a try : http://www.mediafire.com/view/?s3bhc5or15oqxc4
Date issue was solved.
Regarding the 3),
I send you a debug code, Enter that code to cell F7 & hit "Fetch Data".
This will fetch YTD data of all counters, sectors & Indexes.
Process took about 40Sec. on a quad core 8GB system.
It took 35% of processor usage and 60MB or RAM.
This will create some 300+ work sheets.
If you want to delete them use shift key to group select & delete.

kav4 · 16 Re: DFN Data Sniffer Mon Nov 05, 2012 6:34 am

kav4

Senior Manager - Equity Analytics

Posts : 369
Equity Stars : 5262
Reputation : 36
Join date : 2011-06-17

Thanks for your effort & work.

How to get the data for .x0000 shares? (Ex. - SEMB.X0000)

smallville · 17 Re: DFN Data Sniffer Mon Nov 05, 2012 9:04 am

smallville

Associate Director - Equity Analytics

Posts : 4119
Equity Stars : 10129
Reputation : 530
Join date : 2011-02-03
Location : Back on Earth

kav4 wrote:Thanks for your effort & work.

How to get the data for .x0000 shares? (Ex. - SEMB.X0000)

Further more; need data on warrants as well. couldn't find REEF.W19 for an instance..

smallville · 18 Re: DFN Data Sniffer Mon Nov 05, 2012 9:25 am

smallville

Associate Director - Equity Analytics

Posts : 4119
Equity Stars : 10129
Reputation : 530
Join date : 2011-02-03
Location : Back on Earth

Dileepa wrote:

Dileepa wrote:

2) Need an excel formula or a work-around to convert a date in "2009/10/02" format to 091002 (YYMMDD) format..
3) Can you re-code this to fetch data of all counters at once? Also with the option of defining a specific period?

Format issue should be a simple solve. 3) is possible but I think it will increase the load on processor & RAM.
Give me some time and I will give it a try.

Give this a try : http://www.mediafire.com/view/?s3bhc5or15oqxc4
Date issue was solved.
Regarding the 3),
I send you a debug code, Enter that code to cell F7 & hit "Fetch Data".
This will fetch YTD data of all counters, sectors & Indexes.
Process took about 40Sec. on a quad core 8GB system.
It took 35% of processor usage and 60MB or RAM.
This will create some 300+ work sheets.
If you want to delete them use shift key to group select & delete.

By entering the debug code, I got the date in the exact format I wanted Wink

. A simple catch though.. Why is it generating only the 2012 data?

Also can you fetch history data directly from CSE site as well using a similar method.. CSE data is more accurate than broker feeds..

Can this code be modified to get the data in the following format? I only require the following fields in exact order..

DFN Data Sniffer Format10

Thanks a lot for your efforts.. Appreciate..

Dileepa · 19 Re: DFN Data Sniffer Mon Nov 05, 2012 7:31 pm

Dileepa

Senior Manager - Equity Analytics

Posts : 346
Equity Stars : 6273
Reputation : 44
Join date : 2011-03-21
Location : Tokyo

smallville wrote:
By entering the debug code, I got the date in the exact format I wanted . A simple catch though.. Why is it generating only the 2012 data?

Also can you fetch history data directly from CSE site as well using a similar method.. CSE data is more accurate than broker feeds..

Can this code be modified to get the data in the following format? I only require the following fields in exact order..

Thanks a lot for your efforts.. Appreciate..

To fetch full history, it took about 3 min. With all X & W's it will take about 10 min.
There is small risk of accessing DFN DB for longer durations.
If the semaphores are not handled correctly at both ends, that might lead to problems.
Anyway give me some time & I will modify it.
Regarding the data feed, I thought DFN is just parsing the data feed from CSE. Is there a difference?
Also do you want to increment the symbol too (MAL.N00xx?) or is that a typo.

kav4 · 20 Re: DFN Data Sniffer Mon Nov 05, 2012 10:03 pm

kav4

Senior Manager - Equity Analytics

Posts : 369
Equity Stars : 5262
Reputation : 36
Join date : 2011-06-17

Dileepa wrote:

smallville wrote:
By entering the debug code, I got the date in the exact format I wanted . A simple catch though.. Why is it generating only the 2012 data?

Also can you fetch history data directly from CSE site as well using a similar method.. CSE data is more accurate than broker feeds..

Can this code be modified to get the data in the following format? I only require the following fields in exact order..

Thanks a lot for your efforts.. Appreciate..

To fetch full history, it took about 3 min. With all X & W's it will take about 10 min.
There is small risk of accessing DFN DB for longer durations.
If the semaphores are not handled correctly at both ends, that might lead to problems.
Anyway give me some time & I will modify it.
Regarding the data feed, I thought DFN is just parsing the data feed from CSE. Is there a difference?
Also do you want to increment the symbol too (MAL.N00xx?) or is that a typo.

I think it's a typo. He should have drag the value which ended up in incremental value for symbol.

smallville · 21 Re: DFN Data Sniffer Mon Nov 05, 2012 11:39 pm

smallville

Associate Director - Equity Analytics

Posts : 4119
Equity Stars : 10129
Reputation : 530
Join date : 2011-02-03
Location : Back on Earth

Dileepa wrote:

smallville wrote:
By entering the debug code, I got the date in the exact format I wanted . A simple catch though.. Why is it generating only the 2012 data?

Also can you fetch history data directly from CSE site as well using a similar method.. CSE data is more accurate than broker feeds..

Can this code be modified to get the data in the following format? I only require the following fields in exact order..

Thanks a lot for your efforts.. Appreciate..

To fetch full history, it took about 3 min. With all X & W's it will take about 10 min.
There is small risk of accessing DFN DB for longer durations.
If the semaphores are not handled correctly at both ends, that might lead to problems.
Anyway give me some time & I will modify it.
Regarding the data feed, I thought DFN is just parsing the data feed from CSE. Is there a difference?
Also do you want to increment the symbol too (MAL.N00xx?) or is that a typo.

Ah sorry.. its a TYPO... its the N shares and X shares..I accidently dragged the first cell to the rest Mad

The symbols should be like MAL.N0000, PAP.N0000, HNB.X0000... Exactly the way it appears on the CSV file of CSE data..

So if you can automate the entire task cant ask for more.. Very Happy

Is there a possibillity to get data for a date range too?

samueldaniel7 · 22 Re: DFN Data Sniffer Tue Nov 06, 2012 11:25 am

samueldaniel7

Manager - Equity Analytics

Posts : 128
Equity Stars : 5035
Reputation : 33
Join date : 2011-03-21

hello,
can we also get the intraday data
so we can do some rules to find out the short term demands

Dileepa · 23 Re: DFN Data Sniffer Tue Nov 06, 2012 8:08 pm

Dileepa

Senior Manager - Equity Analytics

Posts : 346
Equity Stars : 6273
Reputation : 44
Join date : 2011-03-21
Location : Tokyo

samueldaniel7 wrote:hello,
can we also get the intraday data
so we can do some rules to find out the short term demands

Yes, samu. intraday data also possible & thanks for the MP.
will contact you if I need anything.

Cheers

smallville · 24 Re: DFN Data Sniffer Fri Nov 09, 2012 12:44 pm

smallville

Associate Director - Equity Analytics

Posts : 4119
Equity Stars : 10129
Reputation : 530
Join date : 2011-02-03
Location : Back on Earth

Dileepa.. Any updates?

Dileepa · 25 Re: DFN Data Sniffer Fri Nov 09, 2012 7:29 pm

Dileepa

Senior Manager - Equity Analytics

Posts : 346
Equity Stars : 6273
Reputation : 44
Join date : 2011-03-21
Location : Tokyo

smallville wrote:Dileepa.. Any updates?

Try this : http://www.mediafire.com/view/?ag9fiex7ig9dse0
Changed the format for you.
To get the full history, replace "_YTD" of old debug code with "_ALL"
It will take about 6 min.
Some 30,000 read operations are required to fetch the full history, so be little patient.
Option to specify the date range is no there yet.
Also due to some performance issues, only N's are supported for the time being.
I'm working on those and will let you know once done.

Sponsored content

Name*:
Phone number:
Email address*:
Comments*:
* - required fields	Learn

Name*:
Phone number:
Email address*:
Comments*:
* - required fields	Learn

Latest topics

DFN Data Sniffer

1 DFN Data Sniffer Sat Nov 03, 2012 12:48 pm

Dileepa

2 Re: DFN Data Sniffer Sat Nov 03, 2012 3:06 pm

Jana

3 Re: DFN Data Sniffer Sat Nov 03, 2012 3:22 pm

Dileepa

4 Re: DFN Data Sniffer Sat Nov 03, 2012 3:31 pm

Slstock

5 Re: DFN Data Sniffer Sat Nov 03, 2012 3:47 pm

Dileepa

6 Re: DFN Data Sniffer Sat Nov 03, 2012 4:02 pm

Dileepa

7 Re: DFN Data Sniffer Sat Nov 03, 2012 4:31 pm

Slstock

8 Re: DFN Data Sniffer Sat Nov 03, 2012 5:01 pm

Dileepa

9 Re: DFN Data Sniffer Sat Nov 03, 2012 9:18 pm

kav4

10 Re: DFN Data Sniffer Sat Nov 03, 2012 9:23 pm

Dileepa

11 Re: DFN Data Sniffer Sun Nov 04, 2012 11:27 am

smallville

12 Re: DFN Data Sniffer Sun Nov 04, 2012 11:37 am

smallville

13 Re: DFN Data Sniffer Sun Nov 04, 2012 2:47 pm

Dileepa

14 Re: DFN Data Sniffer Sun Nov 04, 2012 3:06 pm

Dileepa

15 Re: DFN Data Sniffer Sun Nov 04, 2012 5:51 pm

Dileepa

16 Re: DFN Data Sniffer Mon Nov 05, 2012 6:34 am

kav4

17 Re: DFN Data Sniffer Mon Nov 05, 2012 9:04 am

smallville

18 Re: DFN Data Sniffer Mon Nov 05, 2012 9:25 am

smallville

19 Re: DFN Data Sniffer Mon Nov 05, 2012 7:31 pm

Dileepa

20 Re: DFN Data Sniffer Mon Nov 05, 2012 10:03 pm

kav4

21 Re: DFN Data Sniffer Mon Nov 05, 2012 11:39 pm

smallville

22 Re: DFN Data Sniffer Tue Nov 06, 2012 11:25 am

samueldaniel7

23 Re: DFN Data Sniffer Tue Nov 06, 2012 8:08 pm

Dileepa

24 Re: DFN Data Sniffer Fri Nov 09, 2012 12:44 pm

smallville

25 Re: DFN Data Sniffer Fri Nov 09, 2012 7:29 pm

Dileepa

26 Re: DFN Data Sniffer

Sponsored content

New topics