Ivan Idris's Blog, page 20

Secret Transitions of a Markov Chain

Numpy Strategies 0.1.1

Imagine that one day your most important client/Product Manager/Product Owner/Other walks into your room with the following User Story:

I want to make huge profits on the stock market with Markov chains

You think a few seconds and say: "We can implement this easily with NumPy." You split the User Story in Tasks, the first task being determining state transitions and defining the Markov chain model.

A Markov chain is a system that has at least two states. The system switches at random between these states. I would like to define a Markov chain for a stock – AAPL (disclaimer I own AAPL shares). Let's say that we have the states flat F, up U and down D. We can determine the states based on end of day close prices.

1. Obtain 1 year of data

Now we need to obtain the data. One way we can do this is with Matplotlib. This is also mentioned in NumPy Beginner's Guide. We will retrieve the data going back 1 year. Here is the code to do this:

1
2
3
4
today = date.today()
start = (today.year - 1, today.month, today.day)
 
quotes = quotes_historical_yahoo('AAPL', start, today)

2. Select the close price

We now have historical data from Yahoo Finance. The data is represented as a list of tuples, but we are only interested in the close price. We can select the close prices as follows.

1
2
close = [q[4] for q in quotes]
print len(close)

The close price is the fifth number in each tuple. We should have a list of about 253 close prices now.

3. Determine the states

Finally, we can determine the states by subtracting price of sequential days with the NumPy diff function. The state is then given by the sign of the difference. The NumPy sign function returns -1 for a negative, 1 for a positive number or 0 otherwise.

1
states = numpy.sign(numpy.diff(close))

The nice thing about this NumPy code, is that there were very few for loops. I am willing to bet that it is much faster than equivalent "normal" Python code, so it would be nice to measure the difference. Let's leave this as an exercise for the reader.

Next time we will compute the transition probabilities for the various states. We will need these numbers to do fancy things with the stochastic matrix and steady state vector.

If you liked this post and are interested in NumPy check out NumPy Beginner's Guide by yours truly. Last time I checked it was on the bestsellers list of Packt Publishing. This week a review of the book by Marcel Caraciolo generated a lot of buzz on the Internet. Please show your support for this review. I will be back next week with the continuation of the Markov chains story.

6 Great Software Development Books I Own

Last week I showed you a stack of books.

They are all great books that roughly fall in the software development category. I would like to tell you something about them. Some of them I have read many times cover to cover. One of them I actually wrote myself.

1. Introduction to Algorithms

This book is like an encyclopedia of algorithms. The algorithms are presented with pseudo code so it doesn't matter what your favorite programming language is. A very rigorous mathematical approach is used for the analysis of for instance performance.

2. Groovy in Action

Groovy is a new programming language, that is based on Java, but has optional dynamic typing. Groovy also borrows features from Python and Ruby. The examples in this book appealed most to me. Even if, after reading this book as a Java developer, you still want to stick with Java, you would have learned just as much about Java as from any Java book. Groovy is after all very similar to Java.

3. Vi Improved

If you have read my blogs from the vey beginning you would know that I am a Vim fan. Even when I write code in Eclipse I try to do it the vi(m) way as much as possible. I don't recommend reading this book in one go. Rather I suggest reading a bit and then trying out what you read.

4. JUnit Recipes

JUnit is the most popular unit testing framework for Java. JUnit gives you tons of useful unit testing tips. This book is now outdated, since it covers an older version of JUnit – the one without annotations. Still I learned a lot of basic principles from it.

5. Sed & awk

Sed and awk are Unix power tools. Actually Awk is more of a programming language. This book is a good tutorial on both tools. Apparently it is one of the most popular books on the subject.

6. NumPy Beginner's Guide

Finally, it is time to talk about my book NumPy Beginner's Guide. The first and only book for beginners about NumPy. NumPy Beginner's Guide is an action-packed guide for the easy-to-use, high performance, Python based free open source NumPy mathematical library using real-world examples. The book will teach you how to analyze large data sets with statistical functions and execute complex linear algebra and mathematical computations.

This week a new review has come to my attention from Stefan Scherfke – one of the technical reviewers of NumPy Beginner's Guide. Also a few people declared online their intention to review the book – Doug Finke and MousePython. If you are interested in reviewing NumPy Beginner's Guide in return for a free copy have a look at this Python forum post.

Next week I am planning to write a small NumPy example concerning Markov chains. I am looking forward to your visit next week.

NumPy Beginner's Guide Announcement

The long awaited NumPy Beginner's Guide from Packt Publishing is now available in the stores. The reason that I am announcing it here is of course, that I wrote it myself. I just received print copies of the book and I must say it is one of the most beautiful books I have held in my hands.

About the Book

The book is about NumPy and is aimed at people who are interested in learning about NumPy and are able to program with Python. It is a guide filled with practical examples and intentionally low on theory. The examples follow the unique Time For Action tutorial format.

You will learn about:

Installing NumPy
NumPy arrays, matrices and universal functions
NumPy modules
Plotting mathematical NumPy results with Matplotlib
Integration with Scipy, a high level Python scientific computing framework built on top of NumPy

Table of Contents

You can find the complete table of contents on the Packt Publishing website. Please notice how many Time For Action examples there are in this book. They make up the majority of the pages. Here is the list of chapters:

Chapter 1: NumPy Quick Start
Chapter 2: Beginning with NumPy Fundamentals
Chapter 3: Get into Terms with Commonly Used Functions
Chapter 4: Convenience Functions for Your Convenience
Chapter 5: Working with Matrices and ufuncs
Chapter 6: Move Further with NumPy Modules
Chapter 7: Peeking Into Special Routines
Chapter 8: Assure Quality with Testing
Chapter 9: Plotting with Matplotlib
Chapter 10: When NumPy is Not Enough: SciPy and Beyond

Buy the Book

I got a sample of links from the publisher that link to some of the online shops where you can buy the book:

Buy From Safari
Buy From Book Depository
Buy From Barnes and Noble
From TSO shop
Buy From Focus on Training
Buy From Pixtus

If you want to share another link to the book feel free to post it in the comments.

Reviews

I found a mention of the book by Dr. Seth Brown one of the technical reviewers of the book. Let me know, if you want to write a review (so I can link to your review) or reply to this forum thread. The latter guarantees you a free copy of the book.

Acknowledgments

Last but not least, I would like to thank the technical reviewers and the people at Packt Publishing for making this masterpiece possible.

Stackrumban Scrumban StackOverflow style

A thought experiment and Average Price Script 0.0.2

This is a thought experiment, so I have no idea whether it will make sense eventually. Scrumban is a cross between Scrum and Kanban. It has daily meetings but no fixed length sprints. The flow is continuous.

StackOverflow style

This seems very boring to me. Let's add a point system like the one from StackOverflow. Team members can then vote, downvote or accept tasks and associated work. The Hudson build game is based on a similar idea, but it uses automatic scoring. Of course, we want to go further than that and have gradual unlocking of privileges and all sorts of badges.

Average Price Script 0.0.2

I was planning to write a long story about StackOverflow and Stackrumban, but now I decided to await how it all works out. If you are going to make more out of it, remember that I have the copyright! Instead I give the new version of Average Price Script 0.0.1. Now in Python with standard deviation. I used Numpy for the statistics.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
while True:
req = urllib2.Request('http://finance.google.com/finance/inf...' sys.argv[1])
req.add_header('User-agent', 'Mozilla/5.0')
response = urllib2.urlopen(req)
page = response.read()
m = re.search('l_cur" : "(.*)"', page)
prices.append( float( m.group(1) ) )
avg = mean( prices )
sigma = std( prices )
 
devFactor = float(sys.argv[3])
min = avg - devFactor * sigma
max = avg devFactor * sigma
timestr = strftime("%d %b %H:%M:%S", gmtime())
 
print timestr, "Average", avg, numShares(avg), "-Std", min, numShares(min), " Std", max, numShares(max)
time.sleep(int(sys.argv[2]))

Random links of interest

http://quant.stackexchange.com/questions/431/video-lectures-and-presentations-on-quantitative-finance
DukasCopy csv data
Euronext Listed Companies

Average Price Script 0.0.1

So I made a small Perl script again. The script polls data from Google Finance and calculates the average price for a given symbol. The first parameter of the script is the Google Finance symbol that starts with the name of the exchange followed by a colon like 'NYSE:IBM'. The script expects an optional second parameter an integer number indicating the poll interval, 1 by default. Data is retrieved with curl. The response is in the JSON format. I get the current price with a regex from the response. All the prices are stored in an array and averaged. I also do some rudimentary logging.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#!/usr/bin/perl
 
# This script polls data from Google Finance and calculates the average price for a given symbol.
#
# Parameters:
# symbol The Google Finance symbol like 'NYSE:IBM'.
# secondsSleep Integer number indicating the poll interval, 1 by default.
#
$symbol = $ARGV[0];
print "$symbol\n";
 
while(1) {
$data=`curl -s "http://finance.google.com/finance/inf...
$data =~ m/l_cur" : "(.*)"/;
$td=`date '%Y%m%d %H:%M - '`;
push(@prices, $1);
$num = scalar @prices;
$sum = 0;
 
for( $i = 0; $i < $num; $i ) {
$sum = $prices[$i];
}
 
$average = $sum / $num;
print "$num elements, average $average\n";
print "$1 $td";
sleep $ARGV[1];
}

Possible improvements

Here is a list of improvements I have been thinking about.

Use a proper JSON library.
Calculate standard deviation as well.
Calculate a time weighted average, where recent values have a higher weight.
Improve logging.

Random links of interest

List of freely available programming books
GTAC 2010
Irill days 2010

What is the optimal holding period for shares?

Numpy Strategies 0.1.0

Happy New Year! So I was collecting evidence for my prediction of the gold bubble bursting early in 2011 and I found this article. Very short summary – 14 June 2011. What do I think? I think this is completeley incorrect. They are off by at least two days.

The plan for today is:

Define a volatility band for the log of stock prices and mean reverting strategy based on dips below the band.
Optimization for the holding period of shares.
Profit and champagne(optional).

Numpy 4 portfolio closed

The Numpy 4 portfolio was a nice little stable portfolio. I think that it had to do with the Jarque-Bera test prefiltering. Numpy 4 seemed to select mainly energy ETFs. I also did some experimentation with trailing stops. This also might account for its success.

Here is a summary of the closed Numpy portfolios until now.

Stops update

The Numpy 5 portfolio had some startup issues, but this week was pretty OK.

The band

The first step in defining the band is transforming to the logarithm of the stock price. Next, I find the local minima's. For now, I define the local minimum as the point with the lowest value of the set containing this point and 5 earlier and 5 later points.

1
2
3
4
5
6
7
8
9
10
def locMins(arr, period=1):
mins = []
 
for i in range(period,len(arr) - period):
themin = min(arr[i - period : i + period])
 
if arr[i] <= themin:
mins.append(i)
 
return mins

Then, I fit a line through these minima. If the fit fails, Numpy returns an empty residuals array – so I check for that.

1
2
3
4
5
6
7
8
9
10
11
12
logc = log(c[: int(argv[1]) ])
mins = locMins(logc, 5)
 
y = []
 
for m in mins:
y.append( logc[m] )
 
(a,b,residuals) = fitline(mins, y)
 
if len(residuals) == 0:
continue

The band is formed by adding and subtracting the standard deviation of log of the close price with respect to the regression line. I also check whether a certain percentage of the historical prices falls within this band.

1
2
3
4
5
6
7
8
9
def within_band_ratio(arr, a, b, err):
count = 0
 
for i in range( len(arr) ):
if arr[i] <= (a * i + b + err):
if arr[i] >= (a * i + b - err):
count += 1
 
return count / float( len(arr) )

This is what the histogram of the "within band ratio" looks like. Seems to be skewed the right way, don't you agree?

Now it's time for the mean reverting strategy that selects dips below the band.

1
2
3
4
5
6
7
8
9
10
STD = std(logc)
 
last = log( c[ int(argv[1] ) + 1 ] )
if last >= ( a * len(logc) + b - STD):
continue
 
wbr = within_band_ratio(logc, a, b, STD)
 
if wbr < float( argv[2] ):
continue

Optimal holding period

Common sense tells us that the optimal holding period would be less than a day. However, sometimes we don't have enough data for that or whatever. The best choice is of course associated with the largest return, so I try to estimate this. O yeah, I did something with time averaging, ignore it for now.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
...
def expected_return( p, period ):
indices = arange( 0, len(p), period )
q = take( p, indices )
r = get_returns( q )
 
return geomean( r )
 
o,c = loadtxt(path( argv[ 1 ] + '.csv' ), delimiter=',', usecols=(3,6), unpack=True, converters={3: float_converter })
 
ers = []
twers = []
maxEr = expected_return( c, 1 )
ers.append( maxEr )
 
maxTwer = expected_return( c, 1 )
twers.append( maxTwer )
optPeriod = 1
optTWPeriod = 1
 
for i in range(2, 30):
er = expected_return( c, i)
twer = er / i
 
ers.append( er )
twers.append( twer )
 
if er > maxEr:
maxEr = er
optPeriod = i
 
if twer > maxTwer:
maxTwer = twer
optTWPeriod = i
...

Here are the plots of holding periods and expected returns.

This spreadsheet shows the optimal holding periods.

A Vogon Portfolio

Resistance is useless

Prostetnic Vogon Jetzt captain of the yellow constructor ship Grossebanana was feeling a bit sorry for himself. Jetzt denied his crew their shore leave, but that was not enough to cheer him up. Nobody appreciated his poetry. Not even the two hitchhikers, they had just caught.

See, see the sky
Marvel at its big pink depths.
Tell me, do you
Wonder why the goat ignores you?
Why its foobly stare
makes you feel sleepy.
I can tell you, it is
Worried by your shazbaz facial growth
That looks like
A milk.
What's more, it knows
Your fudge potting shed
Smells of frog.
Everything under the big sky
Asks why, why do you even bother?
You only charm cheeses.

It wasn't that bad, was it now? The other problem was that the Prostetnic and the whole Vogon Construction Fleet would be out of work soon. You see, the Vogons were in the business of planet demolition. In the past tearing down planets was necessary to make space for hyperspace travel. But now, with the invention of a new type of spaceship, this was no longer needed. Jetzt was good at two things. Shouting and planet demolition. He had a brilliant career first as a space guard, Junior Shouting Officer, Just Shouting Officer, Senior Shouting Officer, Super Senior Shouting Officer and eventually captain of Grossebanana.

Jetzt did something unusual. He asked the hitchhikers for advice. First, they advised him jokingly to become a pirate, which provoked half hour of the Vogon shouting. When the captain calmed down, the freeloaders asked him whether he had considered trading. That was a fine profession in which shouting was encouraged.

Yes, the Prostetnic thought to himself, that was not such a bad idea. The Vogon's victims encouraged by the sudden silence started talking about some kind of a trading strategy, that was discovered on an ancient planet, known throughout the Galaxy as being "mostly harmless". Unfortunately, soon after the strategy became popular, the planet was demolished by the Vogons. A few hours of shouting later the hitchhikers and Jetzt created the following portfolio:

The captain pulled on the yellow fish stuck in his ear and shouted.

Resistance is useless!

THE END

Random links of general interest

twill
R-Forge TradeAnalytics
R in Finance

If you liked this post and are interested in NumPy check out NumPy Beginner's Guide by yours truly.

How to optimize maximum drawdowns of stock returns

Numpy Strategies 0.0.9

Merry Xmas everybody. If you don't celebrate Xmas, don't worry, you are not missing that much. Maximum drawdown is defined as the maximum decline from a historical peak. Obviously, this is a measure of risk as well. So some optimization is needed. Also I try to shoot below the open price with a new method.

Predictions for 2011

So this is the heathen tradition of trying to forecast the future. It's for one year, so it's 300 times less reliable than the weather forecast. I looked deep into my cup of tea and saw this:

The gold bubble will rise for a short while in 2011, but will burst unexpectedly leading to an uncontrollable fall of the gold price.
The U.S. will invade a country starting with L. The name of the country that is.
A largish meteorite will strike the moon. Nobody would be hurt or injured in any way, but it will lead to outrage and the creation of the Preservation of the Moon Society (PMS).

Resolutions for 2011

So this is the more modern tradition of promising to do things and then forgetting about them after a week.

Blog more often.
Try to keep stock picks private, except of course if in conflict with number 1 resolution.
Improve money management.
Learn a new programming language.
Tell everybody about this programming language and how amazing it is and that everybody should use it. Then tell them again. Repeat until 2012.
What I said in previous bullet points, but substitute "programming language" with technology, tool or whatever.

Numpy 3 closed

The Numpy 3 portfolio was by far the most succesful one. 99% of it created by me and 1% contribution from a grumpy wizard from the Land of Ice. Yes, I could have done the 1 % on my own, but hey that's how it went. We are talking about 17% overall return. It could have been more with tighter stops etc.

Stops update

It was a good week for the Numpy 4 portfolio. No positions had to be closed.

The Numpy 5 portfolio had some startup issues, but this week was pretty OK.

Maximum drawdown optimization

I attempt to optimize the maximum drawdown CAPM style in relation to the Volume Weighted Average Return, skewness of the returns and liquidity.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
...
def maxDrawdown(arr):
top = arr[0]
maxDD = 0
 
for i in range(1, len(arr)):
if arr[i] > top:
top = arr[i]
 
drawdown = (top - arr[i]) / arr[i]
 
if drawdown > maxDD:
maxDD = drawdown
 
return maxDD
...
ev = average(returns, weights=v[:len(returns) ])
maxDD = maxDrawdown( c )
evs.append( ev )
 
maxDDs.append( maxDD )
 
S = stats.skew( returns )
skews.append( S )
 
vol = log( geomean( v ) )
vols.append( vol )
 
if returns[-2] > -1 * float(argv[1]):
continue
 
t = file.replace('.csv', ''), ev, maxDD, S, vol
records.append( t )
 
( a,b,residuals ) = fitline( maxDDs, evs )
( aSkew, bSkew, residuals ) = fitline( maxDDs, skews )
( aVol, bVol, residuals ) = fitline( maxDDs, vols )
 
for t in records:
symbol, evC, maxDD, S, vol = t
 
if evC > a * maxDD b:
if S > 0 and S > aSkew * maxDD bSkew:
if vol > aVol * maxDD bVol:
...

Blind shooting

I try to shoot below the open price with a new method. Starting with the close price of a day, I try extrapolating a low price for the next day. The steps of the extrapolation are:

Calculate relative daily spreads of the close and next day low price.
Estimate the per percentile probability that the projected value will fall within the open – low range.
Use the optimal probability for the actual projection.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
...
tomorL = roll(l,1)
r = (c[:-3] - tomorL[1 :-2])/tomorL[1:-2]
 
N = 1
 
for i in range(1,100):
score = stats.scoreatpercentile(r, float(i) )
p = c[-3]/(1 score)
 
if not isnan(score):
means[i] = score
 
if p >= l[-2] and p < o[-2]:
ratios[i] = 1
 
ratios = ratios / N
...
maxr = max(ratios)
indices = where( ratios >= maxr )
...

Here are some plots based on the data in my historical prices database.

Entries for the Numpy 9 portfolio.

A Xmas portfolio

Bah, humbug!

'Twas the night before Xmas, when all through the house, not a creature was stirring, not even a mouse. Except maybe for old Ibanizar Scruch and he wasn't in a house, but an abandoned office. He had a sweet deal, that allowed him to live in the building, in return he was supposed to keep squatters out. He did that of course unless people paid him enough to let them stay.

Ibanizar wasn't poor. In fact he was quite rich. He had a small business which was a cross between a bank and a hedge fund. A very small one. He jokingly told people that it was a reverse risk neutral Ponzi scheme.

Scruch was a very thrifty fellow. He liked simple living. Ibanizar found the whole idea of spending money on goods and services, that were unnecessary really strange. It seemed to him like the whole world had gone mad. Spending beyond one's means had become the norm. People would only save their money if the interest on a savings account was ridiculously high such as was the case for certain Interweb banks from the Land of Iceland.

The Interweb, that was yet another thing that Scruch despised. Actually he hated technology and in particular information technology. Ibanizar felt that technology only enforced the trend of pursuing the latest and greatest. All the technologists did was repackage old ideas, increase the version of their product and then sell at an ever increasing price. This led to absurdities like companies being worth billions without ever producing anything.

Scruch was fast asleep in an improvised bedlike structure. He made it himself from second hand materials that he purchased from a nearby thrift store. A loud popping noise awoke Ibanizar. He saw a humanlike figure in a white robe standing over him. The creature appeared to be made out of some kind of metal. A blazing light was coming from its head.

"Who are you?" asked Scruch.

"I am the Ghost of Xmas Past," answered the android.

"I don't believe in ghosts," said Ibanizar.

"Do you believe in aliens?"

"Not really. However I don't believe that they don't exist either."

"I am an alien life form. My civillization has evolved in the course of billions of your Earth years, so that we don't need bodies anymore. So we are like ghosts. What you see before you, is just an illusion."

The Ghost of Xmas Past took Scruch back in time. The alien showed the old man how people used to be more frugal and happy in the past. Ibanizar and his childhood friends had almost no toys or possesions. Most of them did not have any allowance. They had lots of fun though, playing and exploring outside. Sometimes their mothers had to drag them back inside. The Ghost transported Ibanizar to the scene of the last Xmas dinner he had with his fiancee Mary. They never ended their relationship, but it just fell apart. She wanted to dance every night, go out, buy a big house, travel all over the world. All those things never interested Scruch. They retained a sort of business relation, that is, he lend her money whenever one of her many business schemes failed.

The Ghost proceeded to show Scruch the four weddings of his former fiancee with increasingly older and richer men. Mary had many kids, lived in many beautiful homes and travelled around the world. She seemed outwardly happy, but was in reality constantly worried and nervous. She constantly worried about money and about the people in her life. Mary smoked a pack of cigarettes a day and her skin had become hard and wrinkled. Scruch led a mindboggingly dull life, but at least he looked like he was still about 30 years old. His emotional barometer pointed to "content", from the moment he woke up in the morning till he went back to bed. Nothing could disturb his peace.

The light on the head of the Ghost of Xmas Past slowly died and Scruch found himself back in his bedroom. Before leaving the Ghost told Ibanizar that he will be visited by two more ghosts. The next ghost took the shape of a giant. He was called the Ghost of Xmas Present.

The giant gave Scruch a tour round the city, showing him all the have and have nots united in their desire to spend, spend, spend. They see a vision of an elderly couple, who borrowed a lot of money during their life. The Andersons spent a large portion of that capital already and the rest was invested in stocks of a large bank. Unfortunately, the bank declared bankruptcy, resulting in an enormous debt. The Ghost of Xmas Present and Scruch also met a guy who slept on a park bench. Ed used to have a good job, but because of his love of alcohol and gambling, he was forced to borrow a lot. At a certain point he had more than a dozen of credit cards. Eventually Ed lost it all and had to sleep on the streets.

The Ghost finally introduces Ibanizar to the Interweb and a blog article about investing. The miser finds the article so interesting that he makes the following virtual Numpy 9 portfolio.

On the stroke of midnight the spirit disappeared and left Scruch to face the Ghost of Xmas Yet to Come. The last Ghost was completely covered by a black robe and appeared to hover. He didn't speak but only pointed with his bony hand. The Ghost takes him to a stockholder meeting in a futuristic setting. Ibanizar had never seen the kind of clothes people are wearing. Nor was he familiar with the tech that surrounded him. Apparently Scruch's company went public after adopting some new trading idea and business was good. So good in fact that Ibanizar has 300 banks as customer. Although that's not much of an achievement since many banks are nationalised, reduced in size or both. Next, Scruch is shown his private island, mansion, jet and yacht. He also has a small submarine, but that is more for showing off, for he hates submarines.

Scruch wakes up very happy and spends the rest of Xmas and the beginning of the next year preparing to execute his new investment plan.

THE END

Random links of general interest

renjin
Apache OpenNLP
mloss
mldata
Impactopia API

If you liked this post and are interested in NumPy check out NumPy Beginner's Guide by yours truly.

Stock returns entropy, the CAPM and target practice

Numpy Strategies 0.0.8

Entropy is a measure of uncertainty and therefore risk. So I attempted to replace the variance of stock returns in the CAPM with entropy. I also tried to save some pennies by "shooting" below the open price on entry. This makes the strategy more selective, because of the possibility that we shoot too low.

Numpy 2 portfolio closed

After six weeks the Numpy 2 portfolio was closed. The expectation was that it would do better than the Numpy 1 portfolio, but it seems that the performance is similar.

Stops update

It was a good week for the Numpy 4 portfolio. Still a few positions had to be closed.

The Numpy 5 portfolio had some startup issues, and this week was also pretty bad.

CAPM and entropy

Entropy is a measure of uncertainty and therefore risk. So I attempted to replace the variance of stock returns in the CAPM with entropy. On the "y" axis I put sequentially the volume weighted average return, skewness of returns and Jarque-Bera pval.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
...
ev = average(returns, weights=v[:len(returns) ])
p,bins = histogram(returns,sqrt(len(returns)),range=(min(returns), max(returns)), normed=True)
entropy = 0
 
for pi in p:
if pi > 0:
entropy += pi * log(pi)
 
 
evs.append( ev )
 
entropies.append( entropy )
 
S = stats.skew( returns )
skews.append( S )
 
jb = jb_pval( returns )
jbs.append(jb)
 
t = file.replace('.csv', ''), ev, entropy, S, jb
records.append( t )
 
 
( a,b,residuals ) = fitline( entropies, evs )
( aSkew, bSkew, residuals ) = fitline( entropies, skews )
( aJb, bJb, residuals ) = fitline( entropies, jbs )
 
for t in records:
symbol, evC, entropy, S, jb = t
 
if evC > a * entropy + b:
if S > aSkew * entropy + bSkew:
if jb > aJb * entropy +bJb:
...

Target practice

In order to save some pennies I try shooting below the open price on entry. It's not much, but it adds up in the long run. It might even be enough to compensate for the transaction costs. This makes the strategy more selective, because of the possibility that we shoot too low. The algorithm requires knowing the open price on the trading day. From this a "low" price is extrapolated using statistical analysis of historical data. The target that we try to hit is the range open to low not including the open price itself. For each security I build up a list of "open low returns". Then I calculate for each percentile of the returns, what the probability is, to hit the target.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
...
r = (o[:-3] - l[:-3])/l[:-3]
 
...
 
for i in range(1,100):
score = stats.scoreatpercentile(r, float(i) )
p = o[-2]/(1 + score)
 
if not isnan(score):
means[i] += score
 
if p >= l[-2] and p < o[-2]:
ratios[i] += 1
...

Below you can see plots for my historical data.

And here are the entries with the CAPM entropy strategy and the shooting method.

A Cardassian portfolio

Gul Voldemordor was the commander of the largest Bajoran workcamp. Before that he was in the Obsidian Order, in charge of gathering information about subversive elements. The Order was one of the best intelligence organizations in the known universe. Some observers said that the average Cardassian could not have a meal without the Order knowing the contents of it.

Obsidians were famous for having a plan within a plan within a plan leading to a trap. The Gul was no exception. He was a very good Kotra player. Another thing he liked was checklists. Voldemordor had a checklist for every occasion.

One day, the camp commander drank a mug of hot fish juice with his breakfast of hardboiled tasper eggs and read his morning checklist:

Punish the Bajorans who underperformed.
Supervise the workers who were building barracks for new arrivals.
Invest the camp funds.

Item 1 appeared regularly on his checklist. It was as if the filthy, lazy Bajorans were trying to sabotage the mining operation. Voldemordor had tried beatings, different forms of torture and cutting off body parts, but nothing seemed to help. As last resort he considered executing some Bajorans to set an example, although he found that a very counterproductive measure.

Item 2 was a recurring theme as well. That was not a problem at all. As a matter of fact the commander welcomed any additions to the workforce. There were concerns raised by some potential Bajoran lovers, but that was all rubbish. The Bajorans were breeding like there was no tomorrow. For every Bajoran sent to the mining planets, ten new Bajorans were born.

Item 3 was his very private thing, that he tried to keep secret. The matter was that business was going rather well. Despite the constant Bajoran sabotage, the sheer number of laborers assured a steady and ever increasing income for the camp. Voldemordor preferred that Central Command did not know this. He cooked the books and reported severe production problems and high operational costs. Of course, he was being watched – primarily by Mineher, the officer in charge of the financial administration department. The Gul had this officer poisoned and then blamed it afterwards on the Bajorans. Three of the Bajoran cooks were shot on the spot.

The Cardassian decided to invest the excess camp income. If the profit was substantial, he would keep it for himself, otherwise he would report it to Central Command. In the latter case it was bound to be interpreted as a sign of his brilliant leadership. The long anticipated promotion would then become a reality. However in the event that there was not enough profit to report – he would just spend the money on a crate of the finest Kanar in the district.

The Gul heard from his finance officer about a dissident on ancient Earth, who had some interesting investment strategies. One of the strategies involved shooting. This appealed to Voldemordor as a military man. Before ordering Mineher to be killed, the commander asked Mineher to help him compose the Numpy 8 portfolio. A purely hypothetical portfolio, of course.

Gul Voldemordor made a new checklist and filed it to be read six weeks into the future:

Sell the stocks on the Numpy 8 portfolio.
Decide what to do with the profit.
Appoint a new, more cooperative financial administration officer.

THE END

Random links of interest

Droplr
Uniquation
S4: Distributed Stream Computing Platform
OpenOpt

If you liked this post and are interested in NumPy check out NumPy Beginner's Guide by yours truly.

How skewed are the prices of stocks?

Numpy Strategies 0.0.7

The 3 moment CAPM takes into account the mean, variance and skewness of asset returns. An investor prefers high positive skewness and low risk, because this corresponds to higher returns. I also did an experiment with fractional Brownian motion.

Numpy 1 Portfolio closure

After six weeks the Numpy 1 portfolio was closed. During this period ULCM merged. The portfolio did rather well, I would say, with more than 50 dollar profit per week.

Stops update

It was a good week for the Numpy 4 portfolio.

The Numpy 5 portfolio had some startup issues, but now everything seems fine.

Fractional Brownian Motion

The fractal brownian motion describes fractal movements. The Hurst exponent H is the main parameter of this model. This exponent has a value between 0 and 1 and is a measure of self similarity of timeseries. A value of 0.5 corresponds with a random walk. H bigger than 0.5 indicates a tendency to trend, while H smaller than 0.5 corresponds with a tendency to oscillate around a mean. H can be estimated from a so called rescaled range. Luckily the Python pyeeg package already has a function that does that.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
...
H = pyeeg.hurst(returns)
 
R = zeros( ( len(c), len(c) ) )
 
for i in range(1, len(c) + 1):
for j in range(1, len(c) + 1):
doubleH = 2 * H
R[i - 1][j - 1] = (i ** doubleH + j ** doubleH - abs(i -j) ** doubleH) / 2
 
sigma = cholesky(R)
path = sigma.dot( standard_normal( len(c) ) ) + c[0]
path = abs(path)
...

Here is a generated chart.

3 moment CAPM

The 3 moment CAPM takes into account the mean, variance and skewness of asset returns. An investor prefers high positive skewness and low risk, because this corresponds to higher returns. I also added the probability of positive returns as an optimization parameter. But that still was not selective enough, so I also select stocks that recently dropped in price by a certain amount.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
...
S = stats.skew( returns )
skews.append( S )
 
pinc = pPos( returns )
pincs.append( pinc )
 
beforeLastReturn = c[ len( c ) - 2 ] / c[ len( c ) - 3 ] - 1
 
if beforeLastReturn > -1 * float( argv[1] ) * ev:
continue
 
t = file.replace('.csv', ''), ev, madC, S, pinc
records.append( t )
 
 
( a,b,residuals ) = fitline( mads, evs )
( aSkew, bSkew, residuals ) = fitline( mads, skews )
( aPinc, bPinc, residuals ) = fitline( mads, pincs )
 
for t in records:
symbol, evC, madC, S, pinc = t
 
if evC > a * madC + b:
if S > 0 and S > aSkew * madC + bSkew:
if pinc > aPinc * madC + bPinc:
...

A Betazoid portfolio

I never met a chocolate I didn't like.

Inikiwi was half human and half Betazoid. Her human father Klojo left her when she was seven, while her mother Imzadi was too busy with her career, so she spent her most of her youth in Federation boarding schools.

Inikiwi's first job as counsellor was on a military starship with a mission to explore deep space. As most Betazoids, Inikiwi was an empath and a real "people person". She did not fit the techie world of a space ship. Inikiwi especially disliked the engineers. They were always scurrying around working on the Perpetuum Mobile Drive. What she found most annoying was the way the techies bumped into her and made insensitive remarks about her shoes.

Inikiwi loved shoes. She loved the sound they made. She loved the way they felt, especially when they were new. In fact she tried to buy as many pairs of shoes as possible. There were weeks, when she would wear a different pair of shoes every single day.

Inikiwi had a pair of Betazed stilleto heels on her wishlist. The heels were so high that made it almost impossible to walk and that's the way the counsellor liked it. It made the engineers doubt all the laws of physics and quantum logic. Not to mention the damage it did to the ship floor.

Unfortunately, such exquisite footwear came with a price. A price, which even a senior Galaxy class captain could not afford. So the only option left was to make some money gambling on the Federation exchange. Inikiwi found an investment consultant, who she felt strongly was trustworthy. He constucted the following portfolio for her:

The securities were bought on a Friday at the open price. All the green numbers made Inikiwi feel warm and fuzzy inside. Inikiwi called this portfolio Numpy 7, for 7 was her lucky number. According to Federation regulation the minimal holding period for securities was six weeks. Inikiwi anticipated more green numbers at that time, so she decided to celebrate in advance with a chocolate cake and chocolate sundae as desert. Yes, she was a true chocolate connoisseur.

THE END

Random links of interest

Historical currency rates
VisualVM plugins
Fractal geometry
Supplemental CS readings

If you liked this post and are interested in NumPy check out NumPy Beginner's Guide by yours truly.

Random walks get you nowhere

Numpy Strategies 0.0.6

Well, you might bump into a wall repeatedly or get hit by a car. The best you can do is get pretty close to your desired destination. So I made some random walk simulations using a trinomial model and a pentanomial model. I also devised a strategy based on a "high band", but that has nothing to do with the random walks.

Stops update

It was a good week for the Numpy 4 portfolio.

The Numpy 5 portfolio had some startup issues.

Trinomial model

According to the trinomial tree model – the stock price can go up, down or stay the same. The logic is undeniable. For each node in the tree we multiply the stock price at that node with a certain factor. This factor is equal to 1, of course if the price is supposed to stay the same. The probabilities for each price change possibility should be calculated using all kinds of parameters. Instead I estimated the probabilities empirically.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
...
def pPos( arr ):
pos = where( array( arr ) > 0 )
return len( pos[ 0 ] ) / float( len( arr ) )
 
def pNeg( arr ):
neg = where( array( arr ) < 0 )
return len( neg[ 0 ] ) / float( len( arr ) )
...
actual = pPos( numbers )
self.assertAlmostEqual( 3/6.0, actual )
 
actual = pNeg( numbers )
self.assertAlmostEqual( 2/6.0, actual )
...
c,v = loadtxt(path( argv[ 1 ] + '.csv' ), delimiter=',', usecols=(6,7), unpack=True)
 
returns = get_returns( c )
 
prices = []
madC = mad( returns )
pincrease = pPos( returns )
pdecrease = pNeg( returns )
n = 2 ** int( argv[ 2 ] )
madSqrt = madC * sqrt(3)
 
for j in range(0, n):
curve = []
curve.append( c[ 0 ] )
rands = uniform(0, 1, len(c))
 
for i in range(1, len( c ) ):
factor = 1
 
if rands[i] < pincrease:
factor = exp( madSqrt )
elif rands[i] < (pincrease + pdecrease):
factor = exp( -1 * madSqrt )
 
curve.append( curve[i - 1] * factor )
 
prices.append( array( curve ) )
 
prices = array( prices )
prices = prices.T
 
 
aggregated = []
 
for curve in prices:
aggregated.append( midhinge(curve) )
...

Notice that I "average" the values for each point in time with midhinge formula.

1
2
3
4
5
6
7
8
9
10
...
def q1(arr):
return stats.scoreatpercentile(arr, 25)
 
def q3(arr):
return stats.scoreatpercentile(arr, 75)
 
def midhinge(arr):
return (q1(arr) + q3(arr)) / 2
...

The resulting plot for AAPL looks a bit boring, but at least we did not end up in a river or something.

Pentanomial tree

The pentanomial tree is an enhancement of the trinomial tree, with two extra levels added.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
...
def pBigUp( arr, limit ):
pos = where( array( arr ) > limit )
return len( pos[ 0 ] ) / float( len( arr ) )
 
def pSmallUp( arr, limit ):
pos = where( array( arr ) < limit )
pos = where( array( take( arr, pos ) ) > 0)
return len( pos[ 0 ] ) / float( len( arr ) )
 
def pBigDown( arr, limit ):
neg = where( array( arr ) < -1 * limit )
return len( neg[ 0 ] ) / float( len( arr ) )
 
def pSmallDown( arr, limit ):
neg = where( array( arr ) > -1 * limit)
neg = where( array( take( arr, neg ) ) < 0)
 
return len( neg[ 0 ] ) / float( len( arr ) )
 
c,v = loadtxt(path( argv[ 1 ] + '.csv' ), delimiter=',', usecols=(6,7), unpack=True)
 
returns = get_returns( c )
 
prices = []
madC = mad( returns )
ev = rgeomean( returns )
limit = ev + 2 * madC
pbu = pBigUp( returns, limit)
pbd = pBigDown( returns, limit )
psu = pSmallUp( returns, limit )
psd = pSmallDown( returns, limit )
n = 2 ** int( argv[ 2 ] )
smallPower = madC
S = stats.skew( returns )
K = stats.kurtosis( returns )
bigPower = smallPower + abs( S ) / (K * 6 )
 
for j in range(0, n):
curve = []
curve.append( c[ 0 ] )
rands = uniform(0, 1, len(c))
 
for i in range(1, len( c ) ):
factor = 1
 
if rands[i] < psu:
factor = exp( smallPower )
elif rands[i] < (psu + psd):
factor = exp( -1 * smallPower )
elif rands[i] < (psu + psd + pbu):
factor = exp( bigPower )
elif rands[i] < (psu + psd + pbu + pbd):
factor = exp( -1 * bigPower )
 
curve.append( curve[i - 1] * factor )
 
prices.append( array( curve ) )
 
prices = array( prices )
prices = prices.T
 
 
aggregated = []
aggregated2 = []
 
for curve in prices:
aggregated.append( (max( curve ) + min( curve )) / 2 )
aggregated2.append( midhinge( curve ) )
...

There seems to be a little bit more action in this plot.

A Borg Portfolio

Resistance is futile.

Her Borg name was 7 of 24. She couldn't remember her real name any more. 7 of 24 thought her new name appropriate, because all she did was work 24/7. The only thing that was promised to her was Borg heaven, where she would be truly assimilated in the Borg. Probably it was because a verse was constantly pumped into her brain.

The Borg believes

that we can achieve

The Borg works 24/7

until we reach heaven

In a previous life Seven was a private detective. In her last year as individual she followed a guy called N.S. who did something with computers. One day, when she was following him something fell out of his pocket. Seven picked it up. It was a piece of paper with some kind of a trading strategy and code scribbled on it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
...
def within_bands_ratio(a,b,sigma,arr):
count = 0
 
for i in range(0, len(arr)):
if(arr[i] < ( a * i + b ) and arr[i] > ( a * i + b - 2 * sigma) ):
count += 1
 
return float(count)/len(arr)
 
for file in get_files():
if(not file.endswith('.csv') ):
print file + 'skipped'
continue
 
c,v = loadtxt(path( file ), delimiter=',', usecols=(6,7), unpack=True)
 
if check_data( c ):
continue
 
returns = get_returns( c )
ev = geomean( returns )
 
beforeLast = len( c ) - 2
indices = where( returns > ev)
tops = take(c[: beforeLast ], indices )
(a, b, residuals) = fitline( indices[0], tops[0] )
 
if len( residuals ) == 0:
continue
 
t = arange(0, len(c), 1)
madC = mad( c )
 
if c[ beforeLast ] < a * beforeLast + b - 2 * madC:
wbr = within_bands_ratio( a, b, madC, c )
 
if wbr > float( argv[ 1 ] ):
...

Seven found it funny, that a computer guy would do something so old school like writing on paper. She reread the note at home several times. The strategy defined a "band" and the idea was to pick stocks that temporarily fell below their band.

The next day she followed N. after he left his home as usual. However, this time he took a completely different route and looked really upset for some reason. N. went into a dark alley and Seven followed. Suddenly the guy turned around, grabbed something from his pocket and pointed it at her. A phaser blast hit Seven.

7 of 24 woke up on a Borg ship, still a bit stunned. A funny verse was playing through her head. Her body was covered with cold pieces of metal. The Borg had the habit of buying slaves cheaply, whenever possible. In these hard economic times, this was becoming increasingly easier.

The planet Rasabanana was also badly hit by the financial crisis. Many Rasabananese lost their homes and some even had to sell themselves as slaves. This was of course a golden opportunity for the Borg, so funds had to be raised to purchase more recruits. The Borg hive sensed a trading strategy floating around in Seven's brain tissue. She was tasked with assembling an investment portfolio.

The Collective had very strict trading regulations. Seven would have to wait at least six weeks until seeing the results. The Borg prepared their standard welcome message for the new drones.

We are the Borg. We will add your biological and technological distinctiveness to our own. Your culture will adapt to service us. Resistance is futile.

THE END

Random links of interest

vimwiki
Pymix
Videolectures
Historical option data
Copperhead
Sqoop