MPT Documentation ./20161219/

Rouwenhorst Sensitivity To js[25,35,45,65,130]/ks[25,35,41,65,130]

MPTrun

index
/Users/katherinepaseman/Documents/projects/Analyst/MPTs/2016 MPT/MPTrun.py

Functions

runMPT20110228Rjk(note, timeFrames, pipelines, js, ks)

runMPT20150922()
#===========================================================================================================

Data

Rpipelines = [{'name': 'plR_m2', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(2); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 2 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m3', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(3); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 3 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m4', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(4); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 4 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m5', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(5); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 5 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m6', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(6); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 6 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m7', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(7); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 7 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m8', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(8); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 8 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m9', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(9); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 9 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}]
i = 9
plRm = 'computeSecurityReturns(); computeRouwenhorstSecu...erCount(%d); computePortfolioReturns(); report();'
shortTimeFrames = ['1 4 2015 12 15 2016']

Analyst.MPT.MPT index
/Users/katherinepaseman/Documents/projects/py/Analyst/MPT/MPT.py

#=========================================================================================================== # Copyright (c) 2006-2015 Paseman & Associates (www.paseman.com). All rights reserved. #=========================================================================================================== SYNOPSIS MPT (Modern Portfolio Theory) lets us write Financial analysis programs in the form of "pipelines". Like unix "pipes" pipelines are straightline programs which process a datastream from quotes to reports. See more documentation in the MPT class. Process Pick a source for historical prices. Default: Yahoo daily o Yahoo - daily, weekly and monthly data, 20 year time horizon. o AAII - monthly data only , 10 year window only. Monthly snaps going back 10 years allows "back testing" and to introduce de-listed tickers, which Yahoo no longer records. o .hmt files - implicit daily data, currently unable to calculate exact dates for the samples. This inhibits some reporting Create Criteria: Each criterion consists of 4 things: - A Name - A Security name set, consisting of a set of ticker symbols. - A Benchmark market index - A Riskless return index E.g. Divide "ticker space" into 4 based on their exchange: American, NYSE, NASDAQ, and OTC o American, Nasdaq and NYSE have fewer "Bad Tickers" than OTC. So we address "selection bias" (http://en.wikipedia.org/wiki/Selection_bias) by partitioning AAII tickers by these exchanges. o Note: We can filter (or partition) the AAII list by any AAII attribute. (../References/20040206 Column Table.xls) Select a set of timeframes o Note: the longer the timeframe, the more "bad tickers" are created Run pipeline for each timeframe, for each criterion, for each ticker set in 3 steps: o Init - Timeframe: --No Default--: Timeframe introduces the greatest variability - Period - (m)onth, (w)eek, (d)ay: Default: "d" o Load - Ticker set(s): Default: American, NYSE, NASDAQ, and OTC - Benchmark index: Default: American- %5EDJI, NYSE- %5EDJI, NASDAQ - %5EIXIC, and OTC - %5ERUA - Riskless Return index: Default: %5EH15_TB_M6 o Run - "j" and "k" lags: Default: [65,130,195,260] days (= 3,6,9,12 months) - Portfolio Count: Default: 10 - Note Prior to 20100516, we deleted items from the middle of the set. Now we default to creating an "unused pool" and report a factorization of the securityName length if the specified portfolioCount does not evenly divide the number of securityNames. Generate Web page with Reports o Running Time + Ticker loading Errors to see that we haven't introduced selection bias o Verification that the test populations have log normal returns. o Rouwenhorst Table I and Table II (including "p"s along with t-test) - returnsTtest = stats.ttest_ind(winnerReturns, indexReturns) - (http://docs.scipy.org/doc/scipy/reference/tutorial/stats.html#comparing-two-samples) - winnerLoserTtest = stats.ttest_rel(winnerReturns, loserReturns) - (http://www.astro.uu.nl/~werkhvn/docs/man/docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_rel.html) TODO o Make Multifactor.py an MPT instruction. o Create Gold Test Set for Rowenhorst, Multifactor and CC. o Debug “I'm sorry, Dave, Im afraid I can't do that.” o src link in html results are off by one character. - listDictAnchorifyValues o 20150120 - screening-for-stock-using-the-lakonishok-approach - o Incorporate AAIIdbf TBD20150320: period is used inconsistently. 'd' is needed for hmtload. 'm' is needed to make ratioToMonthlyReturn match the multifactor baseline # Suggest we remove the 'd' check in load, and use period solely for ratioToMonthlyReturn input. # CHeck if it conflates with J,K. i.e. are j,k speced in days, weeks or months? # So we have three so far: sample period units in Hmt input file; train/test period (j/k) units and reporting period %/mo - %/yr TBD20150917 pass ,_p=p,gapReport=True from MPT.loadSecurityNames to StockDb.loadSecurityNames to loadFromStockhistoryDir into loadFromStockhistoryDir2 (MPTload) MPT.loadSecurityNames(self, src, securityNames, indexSecurityName, risklessReturnSecurityName) StockDb.loadSecurityNames(self, src, securityNames) earlyDate,lateDate = getDates(self.timeFrame) self.securityNames, self.quotes, self.dates = loadFromStockhistoryDir(pathname,securityNames,earlyDate,lateDate,self.period,self) loadFromStockhistoryDir2(pathname,securityNames,earlyDate,lateDate,period,self.p) Future: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2533943 - How validate results? Train. Then measure market α, β, q and p during test period. Question 1: How well does the model eliminate β when p and q match? Would you expect a plot of dividend return rates versus volatility to show an efficient frontier? How "volatile" is the portfolio? i.e. do the portfolio members change drastically with one time step? If so, why? If not, why not? Which "regions" produce the best results, thise with volatile portfolio memebrs or stable ones? Suppose we find a parameter set that biases results into a realm the shows MPT type behavior (e.g. efficient market line). Will MPT style theories work especially well there? Couple Hellmut's Appspot regression analysis to MPT p32 - Graphic 4 - drift with sicherheits puffer Incorporate "Improved Estimates of Correlation Coefficients And Their Impact on the Optimum Portfolios" o Add Sortino ratio; Sharpe Ratio o Add Hellmut's mechanism to calculate optimal quantities o Incorporate "bornDuringInterval/diesDuringInterval" tickers o Fri, Feb 18, 2011 at 11:00 AM - select market segments. E.G. mid caps. Or small caps. Or industries. - 1st day in month - Most Traded - Exchanges o http://matplotlib.sourceforge.net/examples/axes_grid/scatter_hist.html o pull over documentation from below "generateDocumentation" in 20100909MPT_.py FIX QD var(dof) Hellmut p44,53 use sample betas vs. index betas Rouwenhorst with beta+ and beta- with Beta Ranking page 67alpha stratgy for derivatives, p 25 cov = 10.A2 to calculate running Beta for above See also Rp278 to show that buying losers is OK. 4 long; 4 short; sell after 2 mos Per Security Data is kept in dictionaries indexed by securityNames: quotes, securityReturns, securityRankings Ratios Treynor = Ra-Rf/βa T = Treynor ratio, Ri = portfolio i's return, Rf = risk free rate βi = portfolio i's beta Sharpe = (Compound ROR – Rf) / (Standard Deviation of Returns) Sortino = (Compound ROR – Rf) / (Standard Deviation of Negative Returns) #=========================================================================================================== CHANGELOG distinguish debug log, results log (tab file) 20110212 Test Speed Speedup occurs because: o The numpy array routines run primarily in "C". In essence the python interpreter is invoked only 18 times for Rouwenhorst. o We precompute security returns at all lags of interest and reuse them during the computation. This also reduces space requirements. o We only look at Rouwenhorst Winner/Loser, no more intermediate portfolios. Slowdown occurs because: o there is unnecessary calculation at the array level (e.g. the Xl array is zero above membercount, but the numpy array multiplier does not "see" this and so does the multiplication anyway). Note: this version of MPT seems to be about 10x faster than prior versions. Runtime is now dominated by tickerload. Net,Net 20110215 takes about 20 minutes to run. 20100830 used to take more than 3 hours (10x speedup) Using as a base case: Rouwenhorst membercount = 3 DJIA TimeFrame: 1 1 2008 1 1 2010 Period: m Component Count: 30 Compared http://www.paseman.com/projects/Analyst/Hellmut/20100920/20100920RouwenhorstDebug.txt to output of 20100909MPT, 20110212MPTalt and 20110212MPT. Compared Rouwenhorst membercount = 10 AMEX TimeFrame: 1 14 2005 5 14 2008 Period: d Component Count: 254 between 20100909(MPT) and 20110212(MPTalt) Bug 1: ComputeRowenhorstXs used j instead of validFirstIndex, and so gave an index error when used in a CC pipeline Bug 2: Thought ComputeRowenhorstXs set portfolio, changed name to setXsMemberCount and added setXsPortfolioCount 20110214 Compared output of 20100909MPT and 20110212MPT(goldTestMPT) for plCC_p10 j/k=3,3 Bug 1: in 20100909MPT: riskless return needs to be divided by 12*100, not 12 in Bug 2: in 20100909MPT: 1/tau was hardwired to 250 - changed hardwire to 12 for testing Bug 3: Winners/Losers bugs in report 20110217 Bug 1: Hit an interesting bug in Gruber's calculations Some thinly traded stocks maintain the same price for an entire volatility "window" (e.g. CXM). As such, their volatility and average return is zero for that timeframe. so for finite positive Rf, Zi = f((Ri - Rf)/sigma) = - infinity and Xi (= Zi/Sum(Zi)) is garbage. This is what produces some of the NaN (Not a number) results in the 20110215 run. I have altered the code to discard these securities at run time. To simplify my life, I discard the the security for all j's Bug 2: Another source of NaN results is that for small enough values, variance becomes negative (e.g. -1E-17) and taking the square root produces NaN. RunningVariance now takes the absolute value of the result It produces no change in Rouwenhorst Note 3: We can still generate NaN results. Eg. if time frame is a year and volatilty window is also a year, No results are produced and nothng/nothing = NaN 20110218 Bug 1: Russel 200 has time gaps, patched manually in index file Feature 1: Plot R vs. sigma for CC and R. - page 8 20110227-20110301 o Report what stocks to buy today for each strategy o Loop over timeframes to determine timeframe sensitivity o Run tests to see what long/short "numberOfSecuritiesPerPortfolio" works the best - From Sat, May 8, 2010 at 12:34 AM Why do you need 10 deciles? The experience shows that not the deciles but e.g. 4 long 4 short for Dax was optimal. It may be different for other indices. case I understood R. well, then he used these deciles only for looking if there are any kind of peaks in one ranked decile. Literature uses 3rd s also instead of 10 th. The most interesting case is the selection of an optimal number of stocks for going long and optimal number of stocks for going short. If this is true, then you can also use 5 long and 4 short if it is useful e.g. o Run Hellmut's favorite lag - 201412 o Hooked in MPTalt procedures as MPT methods. QD20150320: Discovered the hmtload kept quotes in reverse order

Modules

Analyst.MPT.MPTalt
Analyst.MPT.Multifactor
datetime
numpy
operator
os
pickle
matplotlib.pyplot
re
shutil
scipy.stats
time

Classes



Analyst.StockDb.StockDb

MPT

class MPT(Analyst.StockDb.StockDb)

    MPT is an abstract machine that executes programs having a specialized instruction set. Since MPT is implemented as a "class", multiple different machines can be simultaneously run and compared. The machine's instruction set includes init, analyse, report, computeXXX and setXXX o __init__(self, name, timeFrame, period="d") - timeFrame is the interval of analysis. period is sample period o loadSecurityNames(self, src, securityNames, indexSecurityName, risklessReturnSecurityName) o loadAll(self, src, indexSecurityName, risklessReturnSecurityName) - load security names in the "init" timeframe, and set the market index and riskless return o run(self, pipeline, js, ks) - ultimately generates an in memory datacube of:   [return,j(ranking period),k(holding period),timestep,portfolioNumber]. - js is a list of ranking periods (j's). E.g. [65,130,193,195,260] - ks is a list of holding periods (k's). E.g. [41,65,130,195,260] - "pipeline" is a semicolon delimited string of financial operators. MPT Program Example 1: "computeSecurityReturns(); computeRouwenhorstSecurityRankings(); setXsPortfolioCount(10); computePortfolioReturns(); report();" runs "International Momentum Strategies" -  Rouwenhorst [1998] (../References/JOF/199802_Rouwenhorst.pdf) Quoting Rouwenhorst p269: "The relative strength portfolios are constructed as in Jegadeesh and Titman (1993) (../References/JOF/199303.pdf). At the end of each month, all stocks with a return history of at least 12 months are ranked into deciles based on their past J-month return (J equals 3, 6, 9, or 12) and assigned to one of ten relative strength portfolios (1 equals lowest past performance, or "Loser", 10 equals highest past performance, or "Winner"). These portfolios are equally weighted at formation and held for K subsequent months (K equals 3, 6, 9, or 12 months) during which time they are not rebalanced. The holding period exceeds the interval over which return information is available (monthly), which creates an overlap in the holding period returns. The paper follows Jegadeesh and Titman (1993) who report the monthly average return of K strategies, each starting one month apart. This is equivalent to a composite portfolio in which each month 1/K of the holdings are revised. For example, toward the end of month t the J = 6, K = 3 portfolio of Winners consists of three parts: a position carried over from an investment of one DM at the end of month t - 3 in the 10 percent of firms with highest prior six-month performance as of t - 3, and two similar positions resulting from a DM invested in the top-performing firms at the end of months t - 2 and t - 1. At the end of month t, the first of these holdings will be liquidated and replaced with a unit DM investment in the stocks with highest six-month performance as of time t." MPT Program Example 1: "computeSecurityReturns(); computeCCSecurityRankings_excessReturnToStdDev_Cs(.50); computeCCXlongShorts(); computePortfolioReturns(); report()" runs the long/short computation from Elton/Gruber 6th Ed. p196 PIPELINES Writing code is easy.  Testing, stucturing for reuse and recording results in a consistent way is hard. One way to address these issues is to use "pipelines". "Pipelines" are lists of method calls, similar to straightline programs. Elements ("Stages") of the list are executed in order. Each stage uses data ("Datastores") produced by prior stages in the list. "DataStores" are the only way stages communicate. Data stores (eg securityReturns and portfolioMembers) created by pipeline stages are stored as class members. Each stage optionally returns a dictionary of user readable results. E.g. "report()" creates a text file that reports various statistics on returns. Run returns an aggregate dictionary of all results returned by each stage. To enable pipeline stages integration and reuse, operators keep the data stores consistent. This means that for each pipeline, datastores are named and indexed the same way, e.g.                 | index   |      Array        | securityNames             [securityIndex] securityQuotes            [securityIndex][time] (was quotes          [securityName][t]     ) securityReturns [j or k]  [securityIndex][time] (was securityReturns [securityName][j][t]  ) securityVolatilities[j]   [securityIndex][time] securityRankings[j]       [securityIndex][time] (was securityRankings[securityName][j][t]  ) Xs              [j]   [p] [securityIndex][time] (was portfolios      [j][t][securityIndex] )                                                      memberCounts    [j][t]                                                      portfolioCounts [j][t] {j}{k} rankedReturns             [securityIndex][time] <- now a temporary variable portfolioReturns[j][k][p]                [time] (was portfolioReturns[j][k][t][p]          ) It also means that stages -must- consistently maintain data elements. E.g. returns are monthly, invalid data is signified with -1.00, etc. It does -not- mean that the same datastores are generated by each pipeline. E.g. the datastores below are used for Elton/Gruber portfolio calculation, but not Rouwenhorst. securityVolatilities   [securityName][j][t] securityDesirabilities [securityName][j][t] Pipelines have the following benefits: o Reuse - A single executive(MPT) running multiple scenarios means we use all code we write. o Debugging - Pipelines can be debugged a stage at a time by looking at just the input and output Datastores o Tracking - As data, "Pipelines" and pipeline "stages" can be stored, retrieved, reused, queried and compared.            - THis could not be done if pipelines were implemented as code (e.g. python functions). o Comparison - Running Mulitple pipelines together lets us compare their output. o Programatic Pipeline generation - We can more easily write programs which generate programs   The code below does membercount sensitivity testing on rouwenhorst's algorithm   plRm = "computeSecurityReturns(); computeRouwenhorstSecurityRankings(); setXsMemberCount(%d); computePortfolioReturns(); report();"   Rpipelines = [{"name":"plR_m%d"%i,"stages":(plRm % i),    "text description":"Rouwenhorst ranking, %d members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802" %i} for i in range(2,10)] o Genetic Programming - we can view Pipelines as the "DNA" of an MPT program, which can then be mutated and combined. Note: "Riskless Rate of Return" Unfortunately, Yahoo does -not- seem to have a 6 month (26 week) tbill index.  So we choose the index differently depending on whether we need to pull from the internet directly or from a history directory. When pulling from a history directory, createNewStockHistory.py pulls fed data (Business_day/H15_TB_M6.txt) down, converts it to history format and stores it in %5EH15_TB_M6. When pulling from the internet directly, we use one of the following indices: http://finance.yahoo.com/indices?e=treasury %5ETYX - 30 yr %5ETNX - 10 yr %5EFVX -  5 yr %5EIRX - 13 week (3 month) We assume in all cases that the "Adj Close" column lists the annual rate of return.

Methods defined here:

ConstantCorrelation(self, windowFlag=0, rho=0.5, switches='longShorts', debug=False)

Multifactor(self, memberCount, switch='TreynorRatio', debug=False, compounding='ln')

Rouwenhorst(self, memberCount, debug=False, compounding='ln')
# 20141219

__init__(self, name, timeFrame, period='d', port=None, resultsDir='')
# timeFrame is the interval of analysis (e.g. "1 14 2005 5 14 2008"). # period is the sample period, e.g. (d)ay, (w)eek, (m)onth, (y)ear # port is used to trickle run log status and errors # port is used by p (defined in stockDB) and rp (defined here) # Note that the report method bypasses this port so that reports can go to a separate file

computeCCSecurityRankings_excessReturnToStdDev_Cs(self, rho=0.5)
#=========================================================================================================== # This type of method interface lets us treat methods as "instructions" of an abstract machine # Note instruction dependencies. e.g. setXsMemberCount cannot run until computeSecurityReturns has run. #=========================================================================================================== # requires securityReturns, js, monthlyRisklessReturn # provides securityRankings, excessReturnToStdDev, Cs

computeCCXlongShorts(self, debug=False)
# Follow procedure described in the "Short Sales Allowed" Section in # Elton/Gruber 6th Ed. p198 # a "securityCount" x "timeSlot" mask of all long securities through time that fulfill Gruber's "long short" criteris # requires js, excessReturnToStdDev, Cs, securityVolatilities # provides Xs

computeCCXlongs(self, debug=False)
# Follow procedure described in the "Setting the Cut-off Rate" Section in # Elton/Gruber 6th Ed. p196-197 # Note: Since Rho is always < 1, the only time Cs[0] > excessReturnToStdDev is when securityReturns[j][0] < monthlyRisklessReturn # In this case, memberCountsLong == 0 and ZsLong * maskLong is all zeros # requires js, excessReturnToStdDev, Cs, securityVolatilities # provides Xs

computePortfolioReturns(self)
# requires securityReturns, js, ks # provides portfolioReturns # this is the slowest instruction, about 2x slower than all others combined

computePortfolioReturnsMultifactor(self)
#20150416 - See commented code at bottom of Multifactor.py

computeRouwenhorstSecurityRankings(self)
# requires securityReturns, js # provides securityRankings # argsort(securityReturns,axis=0) sorts securityReturns from smallest to largest along columns (axis=0). # sort would return values, but argsort returns indicies. We reverse indicies order with [::-1]

computeSecurityReturns(self, flag='ln')
# Compute security returns for all js and ks.  Note that # Although j looks back and k looks forward, j and k often use identical returns (e.g. 3,6,9,12 mos) # So we calculate all (monthly) security returns just once here # requires securityQuotes, securityNames, js, ks # provides securityReturns # Use "for" loop to transfer quotes from a dictionary (db friendly) to an array (numpy friendly). # Making the data "numpy friendly" is the key factor that makes MPT 2011 faster than MPT 2010 # securityQuotes' rowIndex keys into securityNames to get securityName # Here, a 9% return on the ith security (Ri) is represented as 1.09.  Elton/Gruber 6th Ed. p45 uses just "9" # Note that E(Ri) and Var(Ri) works for either representation. # We can convert between them using Gruber = 100*(Rowenhorst - 1)

computeSecurityVolatilities(self, windowFlag=0)
# requires securityReturns, js # provides securityVolatilities # modifies securityReturns {Deletes any return row with zero volatility)

dump(self)

finishLoad(self, src, indexSecurityName, risklessReturnSecurityName)
# Contains the common part of loadSecurityNames and loadAll

loadAll(self, src, indexSecurityName, risklessReturnSecurityName)
# unlike loadSecurityNames, we expect component list to contain index and riskless rate of return

loadSecurityNames(self, src, securityNames, indexSecurityName, risklessReturnSecurityName)
# Note: does NOT work for .hmt files unless it contains the index and riskless return histories. # Note: does NOT work for AAII files since they do not even store index or riskless return histories.

maskXsForLastDayOfTheMonth(self)
# "How to beat the market? Only stay a day at a time" - David K. Randall, AP Business Writer, On Sunday January 2, 2011, 2:05 pm EST # http://finance.yahoo.com/news/How-to-beat-the-market-Only-apf-2089752928.html;_ylt=An7scLjTLNPMAd5Ag2czP2Cxba9_;_ylu=X3oDMTFkbG52OTFhBHBvcwMxBHNlYwNuZXdzSHViQXJ0aWNsZUxpc3QEc2xrA2hvd3RvYmVhdHRoZQ--?x=0 # "But a very simplistic form of market-timing has worked for the past 11 years. # It involves owning the Standard & Poor's 500 stocks, but only for the first day of every month."

report(self, switches='WinnerLoser', resultsDir='')

rp(self, title, array, lag=0, fmt='%5.2f')
#rowPrint

run(self, txt, js, ks)
# - js is a list of j's. E.g. [65,130,193,195,260] # - ks is a list of k's. E.g. [41,65,130,195,260] # - txt is a "pipeline" of financial operators. E.g. # "computeSecurityReturns(); computeRouwenhorstSecurityRankings(); computePortfolioMembers(); #  computeMemberAndPortfolioCountsP(10); computePortfolioReturns(); report(); " # "run" ultimately generates an in memory datacube of [return,j(ranking period),k(holding period),timestep,portfolioNumber]. # QD - generate warning if 3,6,8,12 is used with period = w or d # Program statements are separated by semi-colons # whitespace is viewed as arbitrary (so don't us " " in args!)"""

setXsMemberCount(self, memberCount)
# Create long (Xl) and short (Xs) quantities.  Note, depending on k, some X elements may be unused. # Divide the (rank sorted) securities into len(self.securityNames)/memberCount partitions # requires securityReturns, js # provides Xs

setXsPortfolioCount(self, portfolioCount)
# Create long (Xl) and short (Xs) quantities.  Note, depending on k, some X elements may be unused. # Divide the (rank sorted) securities into portfolioCount partitions # requires securityReturns, js # provides Xs

validTimeRange(self, j, k, sampleCount, windowFlag)

Methods inherited from Analyst.StockDb.StockDb:

__getitem__(self, securityName)

idx(self, securityName, date, bisecter=<built-in function bisect_left>)

p(self, s)

printIt(self, options='histories')

removeSuspiciousQuotes(self, factor=10.0, verbose=False)
#=========================================================================================================== # TBD: Need to annualize # Check for stocks that have prices < $0.01 # And of those, check for subsequent prices > $0.00 # Check for stocks that rise/fall > factor in one day. #===========================================================================================================

Functions


array(...)
array(object, dtype=None, copy=True, order=None, subok=False, ndmin=0) Create an array. Parameters ---------- object : array_like     An array, any object exposing the array interface, an     object whose __array__ method returns an array, or any     (nested) sequence. dtype : data-type, optional     The desired data-type for the array.  If not given, then     the type will be determined as the minimum type required     to hold the objects in the sequence.  This argument can only     be used to 'upcast' the array.  For downcasting, use the     .astype(t) method. copy : bool, optional     If true (default), then the object is copied.  Otherwise, a copy     will only be made if __array__ returns a copy, if obj is a     nested sequence, or if a copy is needed to satisfy any of the other     requirements (`dtype`, `order`, etc.). order : {'C', 'F', 'A'}, optional     Specify the order of the array.  If order is 'C' (default), then the     array will be in C-contiguous order (last-index varies the     fastest).  If order is 'F', then the returned array     will be in Fortran-contiguous order (first-index varies the     fastest).  If order is 'A', then the returned array may     be in any order (either C-, Fortran-contiguous, or even     discontiguous). subok : bool, optional     If True, then sub-classes will be passed-through, otherwise     the returned array will be forced to be a base-class array (default). ndmin : int, optional     Specifies the minimum number of dimensions that the resulting     array should have.  Ones will be pre-pended to the shape as     needed to meet this requirement. Returns ------- out : ndarray     An array object satisfying the specified requirements. See Also -------- empty, empty_like, zeros, zeros_like, ones, ones_like, fill Examples -------- >>> np.array([1, 2, 3]) array([1, 2, 3]) Upcasting: >>> np.array([1, 2, 3.0]) array([ 1.,  2.,  3.]) More than one dimension: >>> np.array([[1, 2], [3, 4]]) array([[1, 2],        [3, 4]]) Minimum dimensions 2: >>> np.array([1, 2, 3], ndmin=2) array([[1, 2, 3]]) Type provided: >>> np.array([1, 2, 3], dtype=complex) array([ 1.+0.j,  2.+0.j,  3.+0.j]) Data-type consisting of more than one element: >>> x = np.array([(1,2),(3,4)],dtype=[('a','<i4'),('b','<i4')]) >>> x['a'] array([1, 3]) Creating an array from sub-classes: >>> np.array(np.mat('1 2; 3 4')) array([[1, 2],        [3, 4]]) >>> np.array(np.mat('1 2; 3 4'), subok=True) matrix([[1, 2],         [3, 4]])

backupFiles(template, note, resultsDir)
#=========================================================================================================== # create documentation.html and backup .py files

generateExchangeCriteria(stockDBdir, cap=0.0, DD_A3Mamt=1000.0)
Get all securityNames from AAII database and partition by exchange in which they are listed

generateMPTCube(timeframes, criteria, pipelines, js, ks, period, resultsDir='.')
Generates a results dictionary from the crossproduct of its arguments. "free format" results dictionaries are generated by each pipeline stage and are appended together by "MPT.run". generateMPTCube appends its own result information to a MPT.run's dictionary entry. Runs may take hours and may fail due to program or memory error. generateMPTCube restarts from failure by maintaining a log of intermediate "run" results. When generateMPTCube (re)starts, it skips all cross product elements that have been successfully computed and restarts on the failed element.

generatePlot(resultsDir, filename, results, columnName)
Show risk/return and color by columnName TBD: Vary size of dot by std error of the mean. This would help visualize the true difference between a 2 member portfolio and a 3 member portfolio with the same return. Note: Was unable to get labels to work until I understood that that the model was to call "scatter" once for each "labeled" set of data.  Each of these sets also has its own marker, color and marker size.

generatePortfolioJKStatistics(switches, js, ks, period, sampleCount, windowFlag, portfolioReturns, quotes, indexSecurityName, validTimeRange)
Statistically analyse datacube of (return,j,k,portfolioNumber,timestep) **Usage table1 = generatePortfolioJKStatistics("WinnerLoser",self.js,self.ks,self.period,self.sampleCount,                                 self.portfolioReturns,self.quotes,self.indexSecurityName,self.validTimeRange) **Statistics o Mean/variance are used primarily for their analytical properties, not (necessarily) their results. - Properties include: Var(aX + bY) = Var(X)*a^2 + Var(Y)*b^2 + 2*a*b*Cov(X,Y) - Bienayme''s Formula = Var(Sum(X)) = Sum(Var(X)) o Statistics estimate the parameters of an uncountable population using measurements on a finite number of samples. - Assumption: Data is normally distributed. - We verify that index returns are "log normally distributed" in plotSecurityStatistics o Estimate of population mean = Sum(X)/n. o Estimate of population variance is Sum((X-mean)**2)/n-1. I have a proof for this. o Formula for StdDev resembles the formula for Euclidean Distance.  (modulo sqrt(n-1)) o Standard error of the mean (de Moivre’s equation) is sx = s/root(n). I have a proof for this. o Independent One Sample Test  - t = (XBar - X)/(s/Root(n)) o Dependent Paired Sample Test - t = (XBarA - XBarB)/(s/Root(n)) **Student's t-test http://en.wikipedia.org/wiki/Student%27s_t-test Goal: see if "MPT run" successfully separated the top and bottom portfolios into different populations. Claim: If the mean returns of each population are separated by enough distance, "MPT run pipeline" worked. Method: Use Student's t-test, which is a measure of the distance between two means, corrected for variability. Using Student's t Table, we can determine if these two means come from the same population as a function of: * Degrees of freedom - From Wikipedia's entry on Student's t-test "For significance testing,   the degrees of freedom for this test is 2n − 2 where n is the number of participants in each group."   As such, so long as I run more than 16 samples, I can use df\p row "30" (=2*16 -2). * P - P tells us that the probability of the distance of the mean of returns of winners to the mean of the   losers is zero. 1-p then is the probability that it is not zero or that it is different, so that trading   seems to be sucessful using this strategy.   When P is < 0.0005, I can use the last column of the table.  Coupled with "n", this gives a t of 3.64. **Interpreting Student's t-test If n > 16, t > 3.64 and P < 0.0005, Statistics says that the two means come from two different statistical populations, which means that the method works (post hoc). ttest_ind (independent sample scores) and ttest_rel (related sample scores) comes from "Numerical Recipes" p 483 http://www.scipy.org/doc/api_docs/SciPy.stats.stats.html#ttest_ind This is (presumably) the same as p621 "Numerical Recipes in C++ - 2nd edition" Not yet clear to me yet why ttest_ind(A - B,0) <> ttest_rel(A,B) since Var(A - B) = Var(A) + Var(B) - 2Cov(A,B) Additions made after enabling variable number of memberCounts... Table I reports cases where there is just 1 portfolio in a time slot, but does not include them in the statistics. Table II discards "middle" portfolios in cases like "ConstantCorrelation with C*" when the portfolio count changes from timeSlot to timeSlot.

generateSection(resultsDir, tit, res)

generateWebpage(note, resultsDir, resultsLogFile, timeFrames, criteria, pipelines, js, ks)
Creates the following files: o Index page.  Contains summary figures and links to the pages below. o Datacube Page (from resultsLogFile) - Displays the results log as an Html table and provides a link to an Excel readable version of the log - includes Pipeline section o Program Documentation Page - Contains coalesed Program Documentation o py file backup

zeros(...)
zeros(shape, dtype=float, order='C') Return a new array of given shape and type, filled with zeros. Parameters ---------- shape : int or sequence of ints     Shape of the new array, e.g., ``(2, 3)`` or ``2``. dtype : data-type, optional     The desired data-type for the array, e.g., `numpy.int8`.  Default is     `numpy.float64`. order : {'C', 'F'}, optional     Whether to store multidimensional data in C- or Fortran-contiguous     (row- or column-wise) order in memory. Returns ------- out : ndarray     Array of zeros with the given shape, dtype, and order. See Also -------- zeros_like : Return an array of zeros with shape and type of input. ones_like : Return an array of ones with shape and type of input. empty_like : Return an empty array with shape and type of input. ones : Return a new array setting values to one. empty : Return a new uninitialized array. Examples -------- >>> np.zeros(5) array([ 0.,  0.,  0.,  0.,  0.]) >>> np.zeros((5,), dtype=numpy.int) array([0, 0, 0, 0, 0]) >>> np.zeros((2, 1)) array([[ 0.],        [ 0.]]) >>> s = (2,2) >>> np.zeros(s) array([[ 0.,  0.],        [ 0.,  0.]]) >>> np.zeros((2,), dtype=[('x', 'i4'), ('y', 'i4')]) # custom dtype array([(0, 0), (0, 0)],       dtype=[('x', '<i4'), ('y', '<i4')])

Data

Mpl = "computeSecurityReturns(); computeSecurityVolatil...ePortfolioReturns(); dump(); report('WinnerOnly')"
exp = <ufunc 'exp'>
inf = inf
log = <ufunc 'log'>
loserPortfolioIdx = -1
mgrid = <numpy.lib.index_tricks.nd_grid object>
mpt = <Analyst.MPT.MPT.MPT instance>
pipelines = [{'name': 'plR_p3', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...lioCount(3); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, Three portfolios, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_p10', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...ioCount(10); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, Ten portfolios, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m3', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(3); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, Three members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m10', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...erCount(10); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, Ten members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plCC_p10', 'stages': 'computeSecurityReturns(); computeSecurityVolatil...ioCount(10); computePortfolioReturns(); report();', 'text description': "Constant Correlation ranking, Ten portfolios, 'Winner/Loser' reporting - Elton/Gruber 6thEd p195"}, {'name': 'plCC_XlsW0R40', 'stages': 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();', 'text description': 'Constant Correlation ranking, Two portfolios, Rh... reporting - Elton/Gruber 6thEd p195 j volatility'}, {'name': 'plCC_XlsW0R50', 'stages': 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();', 'text description': 'Constant Correlation ranking, Two portfolios, Rh... reporting - Elton/Gruber 6thEd p195 j volatility'}, {'name': 'plCC_XlsW0R60', 'stages': 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();', 'text description': 'Constant Correlation ranking, Two portfolios, Rh... reporting - Elton/Gruber 6thEd p195 j volatility'}, {'name': 'plCC_XlsW260R40', 'stages': 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();', 'text description': 'Constant Correlation ranking, Two portfolios, Rh...orting - Elton/Gruber 6thEd p195 12 mo volatility'}, {'name': 'plCC_XlsW260R50', 'stages': 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();', 'text description': 'Constant Correlation ranking, Two portfolios, Rh...orting - Elton/Gruber 6thEd p195 12 mo volatility'}, {'name': 'plCC_XlsW260R60', 'stages': 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();', 'text description': 'Constant Correlation ranking, Two portfolios, Rh...orting - Elton/Gruber 6thEd p195 12 mo volatility'}, {'name': 'plCC_XlW0R40', 'stages': "computeSecurityReturns(); computeSecurityVolatil... computePortfolioReturns(); report('WinnerOnly');", 'text description': "Constant Correlation ranking, C* portfolio forma...inner Only' reporting - Elton/Gruber 6thEd p195-6"}, {'name': 'plCC_XlW0R50', 'stages': "computeSecurityReturns(); computeSecurityVolatil... computePortfolioReturns(); report('WinnerOnly');", 'text description': "Constant Correlation ranking, C* portfolio forma...inner Only' reporting - Elton/Gruber 6thEd p195-6"}, {'name': 'plCC_XlW0R60', 'stages': "computeSecurityReturns(); computeSecurityVolatil... computePortfolioReturns(); report('WinnerOnly');", 'text description': "Constant Correlation ranking, C* portfolio forma...inner Only' reporting - Elton/Gruber 6thEd p195-6"}]
plCC = 'computeSecurityReturns(); computeSecurityVolatil...teCCX%s(); computePortfolioReturns(); report(%s);'
plCC_XlW0R40 = "computeSecurityReturns(); computeSecurityVolatil... computePortfolioReturns(); report('WinnerOnly');"
plCC_XlW0R50 = "computeSecurityReturns(); computeSecurityVolatil... computePortfolioReturns(); report('WinnerOnly');"
plCC_XlW0R60 = "computeSecurityReturns(); computeSecurityVolatil... computePortfolioReturns(); report('WinnerOnly');"
plCC_XlsW0R40 = 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();'
plCC_XlsW0R50 = 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();'
plCC_XlsW0R60 = 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();'
plCC_XlsW260R40 = 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();'
plCC_XlsW260R50 = 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();'
plCC_XlsW260R60 = 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();'
plCC_p10 = 'computeSecurityReturns(); computeSecurityVolatil...ioCount(10); computePortfolioReturns(); report();'
plR = 'computeSecurityReturns(); computeRouwenhorstSecu...ioCount(%d); computePortfolioReturns(); report();'
plRR_m10 = 'Rouwenhorst(10); report();'
plRR_m3 = 'Rouwenhorst(3); report();'
plRRm = 'Rouwenhorst(%d); report();'
plR_m10 = 'computeSecurityReturns(); computeRouwenhorstSecu...erCount(10); computePortfolioReturns(); report();'
plR_m3 = 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(3); computePortfolioReturns(); report();'
plR_p10 = 'computeSecurityReturns(); computeRouwenhorstSecu...ioCount(10); computePortfolioReturns(); report();'
plR_p3 = 'computeSecurityReturns(); computeRouwenhorstSecu...lioCount(3); computePortfolioReturns(); report();'
plRm = 'computeSecurityReturns(); computeRouwenhorstSecu...erCount(%d); computePortfolioReturns(); report();'
template = '<html>\n<head>\n<title>%s %s</title>\n\n<script type...1>%s %s</h1><br><h2>%s</h2><br>%s</body>\n</html>\n'
winnerPortfolioIdx = 0

Analyst.MPT.MPTalt index
/Users/katherinepaseman/Documents/projects/py/Analyst/MPT/MPTalt.py

#=========================================================================================================== # Copyright (c) 2006-2014 Paseman & Associates (www.paseman.com). All rights reserved. #=========================================================================================================== This file contains a straightforward implementation of Rouwenhorst, Multifactor and Gruber for testing and exposition. The production version of these algorithms is in MPT, which slices up and shares pieces of the algorithms. Both try to program in an APL style, i.e. few loops, just straight line code using composition and reduction operators on arrays. As in machining or baking cookies, we are given a "sheet" of numbers and we just punch out the right pattern....

Functions


ConstantCorrelation(quotes, dates, securityNames, period, js, ks, monthlyRisklessReturn, windowFlag=0, rho=0.5, switches='longShorts', debug=True)
Trading Strategy described by Elton/Gruber in Modern Portfolio Theory and Investment Analysis 6th Ed pp 195-8 **Usage self.js = self.ks = [3,6,9,12] self.validTimeRange =  validConstantCorrelationTimeRange self.portfolioReturns = ConstantCorrelation(self.quotes,self.dates, array(self.securityNames), self.period,self.js,self.ks,                                             self.quotes[self.risklessReturnSecurityName]/(100 * 12.0))#,True) z = self.report() **Data structure: Approch below lets numpy work easily on the "Array" part                 | index   |      Array        | securityNames             [securityIndex] securityQuotes            [securityIndex][time] securityReturns     [lag] [securityIndex][time] securityVolatilities[j]   [securityIndex][time] {j} excessReturnToStdDev      [securityIndex][time] securityRankings          [securityIndex][time] Cs                        [securityIndex][time] CstarsLong                [securityIndex][time] ZsLong                    [securityIndex][time] XsLong                    [securityIndex][time] ZsLongShort               [securityIndex][time] XsLongShort_Long          [securityIndex][time] XsLongShort_Short         [securityIndex][time] {j}{k} rankedReturns             [securityIndex][time] portfolioReturns[j][k][p]                [time]

Rouwenhorst(quotes, dates, securityNames, period, js, ks, memberCount, debug=False, compounding='ln')
Trading Strategy described by Rouwenhorst in JOF 199802 **Usage self.js = self.ks = [3,6,9,12] self.validTimeRange =  validRouwenhorstTimeRange self.portfolioReturns = Rouwenhorst(self.quotes,self.dates, array(self.securityNames), self.period,self.js,self.ks,3,True) z = self.report() **Number of usable samples is a function of samplesize, j and k. **Example: 24 samples; j,k = 3,6 24 samples <==> sampleCount=24 <==> 0 <= t < 24 <==> range(0,24) eval period (j) = 3 (looking back)      holding period (k) = 6 (looking forward) - securityReturns[3] is 21 long, range(3,24) - securityReturns[6] is 18 long, range(6,24) Rankings start at t = j (t=3) Holding evaluation starts at t = j + k (3 + 6 = 9). Holding evaluation ends   at t = sampleCount - k - 1 (24 - 6 - 1 = 17) If we evaluate the results using securityReturns[k], then the first j (3) samples are unused and the last "k" (6) samples are unused so the total portfolioReturns "t" range is range(j+k, sampleCount - k) or securityReturns[k][j:sampleCount - k] (<> securityReturns[j][k:sampleCount - k] ) So total number of usable samples = sampleCount - (j + 2k) e.g. 24 - (3 + 2*6) = 9 <j>.....................  rankings <-k--><j>.........<-k-->  holdings evaluation 012345678901234567890123 **Data structure: Approch below lets numpy work easily on the "Array" part                 | index   |      Array        | securityNames             [securityIndex] securityQuotes            [securityIndex][time] securityReturns [lag]     [securityIndex][time] {j} securityRankings          [securityIndex][time] Xl                        [securityIndex][time] (vs old Xs[p][securityIndex][time]) Xs                        [securityIndex][time] {j}{k} rankedReturns             [securityIndex][time] portfolioReturns[j][k][p]                [time]  (vs old portfolioReturns[j][k][time][p])

array(...)
array(object, dtype=None, copy=True, order=None, subok=False, ndmin=0) Create an array. Parameters ---------- object : array_like     An array, any object exposing the array interface, an     object whose __array__ method returns an array, or any     (nested) sequence. dtype : data-type, optional     The desired data-type for the array.  If not given, then     the type will be determined as the minimum type required     to hold the objects in the sequence.  This argument can only     be used to 'upcast' the array.  For downcasting, use the     .astype(t) method. copy : bool, optional     If true (default), then the object is copied.  Otherwise, a copy     will only be made if __array__ returns a copy, if obj is a     nested sequence, or if a copy is needed to satisfy any of the other     requirements (`dtype`, `order`, etc.). order : {'C', 'F', 'A'}, optional     Specify the order of the array.  If order is 'C' (default), then the     array will be in C-contiguous order (last-index varies the     fastest).  If order is 'F', then the returned array     will be in Fortran-contiguous order (first-index varies the     fastest).  If order is 'A', then the returned array may     be in any order (either C-, Fortran-contiguous, or even     discontiguous). subok : bool, optional     If True, then sub-classes will be passed-through, otherwise     the returned array will be forced to be a base-class array (default). ndmin : int, optional     Specifies the minimum number of dimensions that the resulting     array should have.  Ones will be pre-pended to the shape as     needed to meet this requirement. Returns ------- out : ndarray     An array object satisfying the specified requirements. See Also -------- empty, empty_like, zeros, zeros_like, ones, ones_like, fill Examples -------- >>> np.array([1, 2, 3]) array([1, 2, 3]) Upcasting: >>> np.array([1, 2, 3.0]) array([ 1.,  2.,  3.]) More than one dimension: >>> np.array([[1, 2], [3, 4]]) array([[1, 2],        [3, 4]]) Minimum dimensions 2: >>> np.array([1, 2, 3], ndmin=2) array([[1, 2, 3]]) Type provided: >>> np.array([1, 2, 3], dtype=complex) array([ 1.+0.j,  2.+0.j,  3.+0.j]) Data-type consisting of more than one element: >>> x = np.array([(1,2),(3,4)],dtype=[('a','<i4'),('b','<i4')]) >>> x['a'] array([1, 3]) Creating an array from sub-classes: >>> np.array(np.mat('1 2; 3 4')) array([[1, 2],        [3, 4]]) >>> np.array(np.mat('1 2; 3 4'), subok=True) matrix([[1, 2],         [3, 4]])

zeros(...)
zeros(shape, dtype=float, order='C') Return a new array of given shape and type, filled with zeros. Parameters ---------- shape : int or sequence of ints     Shape of the new array, e.g., ``(2, 3)`` or ``2``. dtype : data-type, optional     The desired data-type for the array, e.g., `numpy.int8`.  Default is     `numpy.float64`. order : {'C', 'F'}, optional     Whether to store multidimensional data in C- or Fortran-contiguous     (row- or column-wise) order in memory. Returns ------- out : ndarray     Array of zeros with the given shape, dtype, and order. See Also -------- zeros_like : Return an array of zeros with shape and type of input. ones_like : Return an array of ones with shape and type of input. empty_like : Return an empty array with shape and type of input. ones : Return a new array setting values to one. empty : Return a new uninitialized array. Examples -------- >>> np.zeros(5) array([ 0.,  0.,  0.,  0.,  0.]) >>> np.zeros((5,), dtype=numpy.int) array([0, 0, 0, 0, 0]) >>> np.zeros((2, 1)) array([[ 0.],        [ 0.]]) >>> s = (2,2) >>> np.zeros(s) array([[ 0.,  0.],        [ 0.,  0.]]) >>> np.zeros((2,), dtype=[('x', 'i4'), ('y', 'i4')]) # custom dtype array([(0, 0), (0, 0)],       dtype=[('x', '<i4'), ('y', '<i4')])

Data

inf = inf
loserPortfolioIdx = -1
mgrid = <numpy.lib.index_tricks.nd_grid object>
sqrt = <ufunc 'sqrt'>
winnerPortfolioIdx = 0

Analyst.MPT.Multifactor index
/Users/katherinepaseman/Documents/projects/py/Analyst/MPT/Multifactor.py

#=========================================================================================================== # Copyright (c) 2006-2015 Paseman & Associates (www.paseman.com). All rights reserved. #=========================================================================================================== # HTC: is "Hellmut Test Case" for DAX_20141014.hmt in projects/Analyst/Analyst\ Hellmut/20141115\ Hellmut\ -\ Is\ Systematic\ Risk\ diversifiable/Multifactor.py

Modules

numpy

Functions

Multifactor(quotes, dates, securityNames, indexSecurityName, period, js, ks, memberCount, switch='TreynorRatio', debug=False, compounding='ln')
Trading Strategy described by Scholtz 201412 Based on Analyst/Analyst Hellmut/20141115 Hellmut - Is Systematic Risk diversifiable/20141213Multifactor.py **Usage self.js = [193] training periods self.ks = [42] test periods self.portfolioReturns = Multifactor(self.quotes,self.dates, np.array(self.securityNames),self.indexSecurityName self.period,self.js,self.ks,3,True) z = self.report() The rest of the algorithm is similar to Rouwenhorst

calcTerm(BetaMasked, n, securityReturns, securityRankings, window, mask)
#===========================================================================================================

maskBeta(indexReturns, securityReturns, window, mask)
#===========================================================================================================

selectRankedElements(anArray, rowRankings)
#=========================================================================================================== # np.argsort returns the security rankings for each timestep. # selectRankedElements ranks each column .

trainingDataToWeights(indexReturns, securityReturns, portfolioIdxes, window, switch='TreynorRatio')
#===========================================================================================================

Data

loserPortfolioIdx = -1
winnerPortfolioIdx = 0

Analyst.MPTutil index
/Users/katherinepaseman/Documents/projects/py/Analyst/MPTutil.py

Modules

numpy

Functions


annualised_Sharpe(daily_ret, yearly_benchmark_rate=0.05, N=252)

annualised_Sortino(daily_ret, yearly_benchmark_rate=0.05, N=252)

array(...)
array(object, dtype=None, copy=True, order=None, subok=False, ndmin=0) Create an array. Parameters ---------- object : array_like     An array, any object exposing the array interface, an     object whose __array__ method returns an array, or any     (nested) sequence. dtype : data-type, optional     The desired data-type for the array.  If not given, then     the type will be determined as the minimum type required     to hold the objects in the sequence.  This argument can only     be used to 'upcast' the array.  For downcasting, use the     .astype(t) method. copy : bool, optional     If true (default), then the object is copied.  Otherwise, a copy     will only be made if __array__ returns a copy, if obj is a     nested sequence, or if a copy is needed to satisfy any of the other     requirements (`dtype`, `order`, etc.). order : {'C', 'F', 'A'}, optional     Specify the order of the array.  If order is 'C' (default), then the     array will be in C-contiguous order (last-index varies the     fastest).  If order is 'F', then the returned array     will be in Fortran-contiguous order (first-index varies the     fastest).  If order is 'A', then the returned array may     be in any order (either C-, Fortran-contiguous, or even     discontiguous). subok : bool, optional     If True, then sub-classes will be passed-through, otherwise     the returned array will be forced to be a base-class array (default). ndmin : int, optional     Specifies the minimum number of dimensions that the resulting     array should have.  Ones will be pre-pended to the shape as     needed to meet this requirement. Returns ------- out : ndarray     An array object satisfying the specified requirements. See Also -------- empty, empty_like, zeros, zeros_like, ones, ones_like, fill Examples -------- >>> np.array([1, 2, 3]) array([1, 2, 3]) Upcasting: >>> np.array([1, 2, 3.0]) array([ 1.,  2.,  3.]) More than one dimension: >>> np.array([[1, 2], [3, 4]]) array([[1, 2],        [3, 4]]) Minimum dimensions 2: >>> np.array([1, 2, 3], ndmin=2) array([[1, 2, 3]]) Type provided: >>> np.array([1, 2, 3], dtype=complex) array([ 1.+0.j,  2.+0.j,  3.+0.j]) Data-type consisting of more than one element: >>> x = np.array([(1,2),(3,4)],dtype=[('a','<i4'),('b','<i4')]) >>> x['a'] array([1, 3]) Creating an array from sub-classes: >>> np.array(np.mat('1 2; 3 4')) array([[1, 2],        [3, 4]]) >>> np.array(np.mat('1 2; 3 4'), subok=True) matrix([[1, 2],         [3, 4]])

debugLogAdd(debugLog, name, value, j)
#===========================================================================================================

ratioToMonthlyReturn(ratio, lag, period, flag='ln')
Convert finalCapital[t]/initialCapital[t-lag] ratios into monthly returns, as in Rouwenhorst. 'lag' is in units of 'period'. The function looks up 1/tau: a constant = 252, 52, 12, or 1 for 'day', 'week', 'month' or year respectively. If flag <> 'ln', ratioToMonthlyReturn divides 1/tau by 12 times the lag and multiplies it by the input ratio minus one. E.g. Suppose a stock goes from 100 to 105 in 3 months.  It has earned 5% in those 3 months. To determine monthly return, we run: >>> ratioToMonthlyReturn(105.00/100.00 ,3.0 , "m","ratio") 0.01666666666666668 This produces 0.01666 since this = 1.666%/month Note1: return = (ratio - 1.0) * bias = ratio * bias - bias;  ratio can get arbitrarily close to zero, so "bias" is the minimum number we can add to the worst case "return" to make it positive. if flag == 'ln' >>> ratioToMonthlyReturn(105.00/100.00 ,3.0 , "m") 0.0162633880565 Note2: ratio my be input as a scalar or a numpy array. >>> ret = array([0.5, 0.93, 0.95, 0.98, 1.00, 1.02, 1.04, 1.06, 1.5]) >>> ratioToMonthlyReturn(ret ,3.0 , "m") array([-0.23104906, -0.02419023, -0.01709776, -0.00673424,  0.  0.00660088,  0.01307357,  0.01942297,  0.13515504]) >>> ratioToMonthlyReturn(ret ,3.0 , "m", "ratio") array([-0.16666667, -0.02333333, -0.01666667, -0.00666667,  0.  0.00666667,  0.01333333,  0.02      ,  0.16666667])

rowFormat(title, array, lag=0, fmt='%5.2f')
Generic numpy array print routine print each array (or vector)'s title followed by each row, indented by lag and each cell formated with fmt

rowPrint(title, array, lag=0, fmt='%5.2f')

runningLinearRegression(x, y, window, includeAlpha=False, n=[], debug=False)
# Beta(x,y) = cov(x,y)/var(x) # Alpha(x,y) = mean(y) - Beta(x,y)*mean(x) # The trick with nullValues is to make sure that the beta/alpha results appear in the correct positions. # The caller calculates n (when it is a vector) since the caller uses it for other stuff as well.

runningSharpeOrSortinoRatio(excess_returns, window, period, sortino, debug=False)
#https://www.quantstart.com/articles/Sharpe-Ratio-for-Algorithmic-Trading-Performance-Measurement Calculate the annualised Sharpe ratio of a returns stream based on a number of trading periods, window. window is often 193 or 42, which then assumes a stream of daily returns. The function assumes that the returns are the excess of those compared to a benchmark. # http://investexcel.net/how-to-annualize-a-sharpe-ratio/ Sharpe Ratios are equal to the effective return divided by the standard deviation. Commonly, Sharpe Ratios on a daily, weekly or monthly basis are annualized by multiplying by the square root of the higher frequency time period. This is because the effective return is proportional to time. Assuming a Weiner process governs stock prices, variance is proportional to time. Hence standard deviation is proportional to the square root of time. So you would scale a Sharpe Ratio by multiplying by t/√t = √t, where t is the frequency you are annualizing from. To summarize, Monthly Sharpe Ratios are annualized by multiplying by √12 Daily Sharpe Ratios are annualized by multiplying by √252 (assuming 252 trading days in a year) But (and this is a big but), a paper has demonstrated that this is misleading, and can often overestimate the actual Sharpe Ratio. The Sharpe Ratio is calculated from estimated quantities, and subject to errors. You have to take into account the specific properties of the returns distribution. For example, the Sharpe Ratio of hedge funds can be overestimated by 65% or more if you do not correctly model their serial autocorrelation. #http://www.investopedia.com/video/play/sharpe-ratio/ # >1 is good >3 is excellent # http://finance.yahoo.com/q/rk?s=SPY - Gets a sharpe ratio from Google #https://www.quantopian.com/posts/calculating-sharpe-ratio-2 #sharpe = (data[stock].returns() / np.std(data[stock].returns())) * np.sqrt(365.25)   !!!!! This is wrong.  Final term ought be * np.sqrt(252.0)

runningSharpeRatio(excess_returns, window, period, debug=False)
#=========================================================================================================== # FRACTIONAL. NOT IN %.  TO GET %, MULTIPLY RESULT BY 100!

runningSortinoRatio(excess_returns, window, period, debug=False)

runningVariance(x, window, n=[], debug=False)
Create a variance array in O(len(x)). TBD: use http://pandas.pydata.org/pandas-docs/stable/computation.html#moving-rolling-statistics-moments moving, rolling, window, incremental http://code.google.com/p/incremental-statistics/ http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Two-pass_algorithm equivalent to   return array([var(x[:,t:t+n],1) for t in range(1+x.shape[1]-n)]).T if we weighted var with n/n-1 Hellmut used this approach to create a variance array in O(len(x)) rather than O(len(x)*n) by using two running sums as intermediate variables. See Mon, Aug 16, 2010 at 4:42 Aggregate variance of last n samples into current data point. E.g. prices = array([1.0,10,100,1000,10,1000,100000,10000,100000,1000000,10000000,100000,10000000,1000000000,100000000]) returns = pad(  [0],  log10(prices[1:]/prices[:-1]) ) runningVariance(returns, 2.0) z = array([0, 1,  1,  1, -2,  2,  2, -1,  1,  1,  1, -2,  2,  2, -1 ]) zz = vstack((z,z)) print zz                       [0, 1,  1,  1, -2,  2,  2, -1,  1,  1,  1, -2,  2,  2, -1 ]                       [0, 1,  1,  1, -2,  2,  2, -1,  1,  1,  1, -2,  2,  2, -1 ] print runningVariance(zz, 2, True)                          [0.5 0.  0. 4.5  8.  0. 4.5  2.  0.  0. 4.5  8.  0. 4.5]                          [0.5 0.  0. 4.5  8.  0. 4.5  2.  0.  0. 4.5  8.  0. 4.5]                        E.g.           /\ (1 - -.5)**2 + (-2 - -.5)**2 = 4.5

test_sharpe()
#-----------------------------------------------------------------------------------------------------------

test_sortino()
#-----------------------------------------------------------------------------------------------------------

validFirstIndex(j, window=0)
Rouwenhorst looks at j-lagged returns, e.g. for j=3, P[3]/P[0], P[4]/P[1], P[5]/P[2] ... P[n-1]/P[n-4] Window refers to the time window used in the variance calculation by Constant Correlation. There are generally two cases here.  First is that we have a window of one year. Second is that we have a window = j. j since securityReturns[j] is missing j values. window, since volatility is missing window-1 values, but is based on returns, which adds 1 back. Note: Actually, we assume window == 0, or window > j

volatility(returns, window, period)
"3 month volatility" is the volatility over a 3 month window of 1 month returns (NOT the volatility over a 3 month window of 3 month returns). So if Period = "d", returns used in the volatility calculation are ln(P[day n]/P[day n-1])    If Period = "m", returns used in the volatility calculation are ln(P[month n]/P[month n-1]) Hull 4th Edition 11.3 p242; Hellmut's Edition 4.5 p88 says the following: 1) Volatility is -dependent- on sample interval. -   Volatility = sigma * sqrt(1/tau) - Hull lists 1/tau = 252 for daily samples (presumably 52 for weekly samples, 12 for monthly samples, 1 for yearly samples?) 2) Volatility is -independent- of sample size, but its standard error is -dependent- on sample size. -   StdErr = volatility/sqrt(2*n) For a column of 20 daily values, Hull calculates (- in percent-) YearlyVolatility=100*STDEVA(F4:F23)*SQRT(252) StandardError=YearlyVolatility/SQRT(2*20) For a column of 12 monthly values YearlyVolatility=100*STDEVA(F37:F39)*SQRT(12) or YearlyVolatility=100*STDEVA(F28:F39)*SQRT(12) and for a column of 252 daily values YearlyVolatility=100*STDEVA(F236:F296)*SQRT(252) or YearlyVolatility=100*STDEVA(F45:F297)*SQRT(252) Tuesday, August 17, 2010 10:29 AM in response to Hull Hull: "An often used rule of thumb is to set the time period over which the volatility is measured equal to the time period over which it is to be applied. Thus, if the volatility is to be used to value a two year option, two years of historical data are used." Hellmut: 1) "application" is the holding period and 2) volatility of period k = (vola j) * sqr(k)/sqr(j). So the difference between 3 month volatility and yearly volatility estimated from a 3 month sample. sqrt(3/12) for monthly samples and sqrt(62/252) for daily samples Check with Hellmut: Assume that 1/tau = f(data measurement period) and we want "per month", not "per annum" Note 1: Elton/Gruber 6th Ed. p195 uses variance, not volatility. Note 2: Since the key calculation is (E(Ri)-Rf)/variance, Ri, Rf and variance ought be measured over the same timeframe. Note 3: Elton/Gruber 6th Ed. p45 uses N vs. N-1 to obtain a maximum likelihood estimate. Note 4: "N vs. N-1" and volatility vs. variance won't matter in ranking, but does affect Z.

zeros(...)
zeros(shape, dtype=float, order='C') Return a new array of given shape and type, filled with zeros. Parameters ---------- shape : int or sequence of ints     Shape of the new array, e.g., ``(2, 3)`` or ``2``. dtype : data-type, optional     The desired data-type for the array, e.g., `numpy.int8`.  Default is     `numpy.float64`. order : {'C', 'F'}, optional     Whether to store multidimensional data in C- or Fortran-contiguous     (row- or column-wise) order in memory. Returns ------- out : ndarray     Array of zeros with the given shape, dtype, and order. See Also -------- zeros_like : Return an array of zeros with shape and type of input. ones_like : Return an array of ones with shape and type of input. empty_like : Return an empty array with shape and type of input. ones : Return a new array setting values to one. empty : Return a new uninitialized array. Examples -------- >>> np.zeros(5) array([ 0.,  0.,  0.,  0.,  0.]) >>> np.zeros((5,), dtype=numpy.int) array([0, 0, 0, 0, 0]) >>> np.zeros((2, 1)) array([[ 0.],        [ 0.]]) >>> s = (2,2) >>> np.zeros(s) array([[ 0.,  0.],        [ 0.,  0.]]) >>> np.zeros((2,), dtype=[('x', 'i4'), ('y', 'i4')]) # custom dtype array([(0, 0), (0, 0)],       dtype=[('x', '<i4'), ('y', '<i4')])

Data

abs = <ufunc 'absolute'>
log = <ufunc 'log'>
loserPortfolioIdx = -1
maxBias = 21.0
minimum = <ufunc 'minimum'>
oneOverTauTable = {'d': 252.0, 'm': 12.0, 'w': 52.0, 'y': 1.0}
sqrt = <ufunc 'sqrt'>
winnerPortfolioIdx = 0

Analyst.MPTload index
/Users/katherinepaseman/Documents/projects/py/Analyst/MPTload.py

Modules

numpy
os

Functions


bisect_left(...)
bisect_left(a, x[, lo[, hi]]) -> index Return the index where to insert item x in list a, assuming a is sorted. The return value i is such that all e in a[:i] have e < x, and all e in a[i:] have e >= x.  So if x already appears in the list, i points just before the leftmost x already there. Optional args lo (default 0) and hi (default len(a)) bound the slice of a to be searched.

bisect_right(...)
bisect(a, x[, lo[, hi]]) -> index bisect_right(a, x[, lo[, hi]]) -> index Return the index where to insert item x in list a, assuming a is sorted. The return value i is such that all e in a[:i] have e <= x, and all e in a[i:] have e > x.  So if x already appears in the list, i points just beyond the rightmost x already there Optional args lo (default 0) and hi (default len(a)) bound the slice of a to be searched.

loadFromStockhistoryDir2(pathname, securityNames, earlyDate, lateDate, period, _p, gapReport=False, maxAllowableMissingEntries=50)
#===========================================================================================================

p(s)
#===========================================================================================================

Data

DAXcomponents = ['ADS.DE', 'ALV.DE', 'BAS.DE', 'BAYN.DE', 'BEI.DE', 'BMW.DE', 'CBK.DE', 'CON.DE', 'DAI.DE', 'DB1.DE', 'DBK.DE', 'DPW.DE', 'DTE.DE', 'EOAN.DE', 'FME.DE', 'FRE.DE', 'HEI.DE', 'HEN3.DE', 'IFX.DE', 'LHA.DE', ...]

Analyst.StockDb index
/Users/katherinepaseman/Documents/projects/py/Analyst/StockDb.py

#=========================================================================================================== # Copyright (c) 2006-2011 Paseman & Associates (www.paseman.com). All rights reserved. #=========================================================================================================== # StockDb returns an array of quotes for each of a specified list of securityNames for a timeFrame. # # **Dirty Data** # AAII <> Yahoo # AAII and Yahoo give different dieDuringInterval values for the same interval!!! # This is to be expected, since each increases prior prices by the amounts of intervening dividends. # So prior AAII dbs of quotes are totally out of sync with Yahoo's current price set. # TBD: get Yahoo -monthly- snap at same time AAII does and compare AAII to Yahoo values. # # **Selection Bias** # This module can introduce selection bias via "Incomplete Histories" and "Suspicious Quotes". # # >>>Incomplete Histories # Of the following cases, we are most concerned about the first one: # - Quote histories start before beginning of timeframe and end before end of timeframe # - Quote histories start before beginning of timeframe and end after end of timeframe # - Quote histories start after beginning of timeframe # - Quote histories start after end of timeframe # - Quote histories end before beginning of timeframe # StockDb reports tickers with incomplete histories over the requested intervals. # Those which come into existence during the interval (bornDuringInterval) are OK. # Those which end during the interval (diedDuringInterval) are discarded by the code, so there is a potential to undercount losses. # # >>>Suspicious Quotes # Quotes which reach zero over their lifetime, or drop/rise by a large factor in one period (day). # moving return factor from 5x drop to 10x drop allowed all NYSE stocks to pass. # American stocks: IVD has spike in 2008. # WAC went from 0.13 to 5.30 with zero shares traded. # # Limiting by exchanges is the best way to address both incomplete histories and Suspicious Quotes. # Below is a run for the four AAII exchange classification: American, NASDAQ, OTC and NYSE for downloaded histories: # OTC has the most anomolies and Nasdaq, NYSE and American the least # # o Step 1 - alert to the condition # o Step 2 - narrow selection criteria so that we have few companies without a complete history # - Will not account for M&A, e.g. Acquisition of Shering-Plough # o Step 3 - keep selection criteria wide, but account for new stocks coming in, or old stocks going out # - Lot of Coding! # TBD: (Somehow) incorporate prior implementation which kept quotes as a dictionary, keyed by time

Classes

StockDb

class StockDb

Most portfoio analysis and construction algorithms require a database of stock info (primarily quotes) to operate. # StockDb holds a dictionary of these histories keyed by ticker, plus several other variables. # Once created, a StockDb is read-only # It can be initialized from a variety of sources: # o A .hmt file. src is a string ending in ".hmt" # o The yahoo quote service url. src is a list # o A stockhistory directory containing yahoo quote files. src is a string containing "stockhistory" # filenames are of the form "ticker.csv", e.g. "AA.csv" for Alcoa # o Another ticker StockDb - Note that we shallow copy histories to save space. # o "AAII" directory - contains Corporate information. see aaii.py. src is a string containing "AAII" # # Unlike documents/py/autoanalyst/StockDB.py, we store "tickers" as a list, not dictionary keys since order matters. # Both the "AAII" and "history" directories structure data using "DateStampedPath" organization: see history.py # # Members: # StockDb.name # StockDb.timeFrame # StockDb.period # StockDb.quoteImportTime # StockDb.securityNames = [] # StockDb.quotes[securityName] = quoteHistoryArray # StockDb.dates # StockDb.sampleCount #===========================================================================================================

Methods defined here:

__getitem__(self, securityName)

__init__(self, name, timeFrame, period='m', port=None)

idx(self, securityName, date, bisecter=<built-in function bisect_left>)

loadAll(self, src)
#=========================================================================================================== # Load all securityNames from source into StockDb # Works for .hmt, stockhistory dir, AAII dir, yahoo internet, AAII db, StockDb #===========================================================================================================

loadSecurityNames(self, src, securityNames)
#=========================================================================================================== # Load selected securityNames from source into StockDb # Works for stockhistory dir, AAII dir, yahoo internet, AAII db - not for .hmt, StockDb #===========================================================================================================

p(self, s)

printIt(self, options='histories')

removeSuspiciousQuotes(self, factor=10.0, verbose=False)
#=========================================================================================================== # TBD: Need to annualize # Check for stocks that have prices < $0.01 # And of those, check for subsequent prices > $0.00 # Check for stocks that rise/fall > factor in one day. #===========================================================================================================

Functions


array(...)
array(object, dtype=None, copy=True, order=None, subok=False, ndmin=0) Create an array. Parameters ---------- object : array_like     An array, any object exposing the array interface, an     object whose __array__ method returns an array, or any     (nested) sequence. dtype : data-type, optional     The desired data-type for the array.  If not given, then     the type will be determined as the minimum type required     to hold the objects in the sequence.  This argument can only     be used to 'upcast' the array.  For downcasting, use the     .astype(t) method. copy : bool, optional     If true (default), then the object is copied.  Otherwise, a copy     will only be made if __array__ returns a copy, if obj is a     nested sequence, or if a copy is needed to satisfy any of the other     requirements (`dtype`, `order`, etc.). order : {'C', 'F', 'A'}, optional     Specify the order of the array.  If order is 'C' (default), then the     array will be in C-contiguous order (last-index varies the     fastest).  If order is 'F', then the returned array     will be in Fortran-contiguous order (first-index varies the     fastest).  If order is 'A', then the returned array may     be in any order (either C-, Fortran-contiguous, or even     discontiguous). subok : bool, optional     If True, then sub-classes will be passed-through, otherwise     the returned array will be forced to be a base-class array (default). ndmin : int, optional     Specifies the minimum number of dimensions that the resulting     array should have.  Ones will be pre-pended to the shape as     needed to meet this requirement. Returns ------- out : ndarray     An array object satisfying the specified requirements. See Also -------- empty, empty_like, zeros, zeros_like, ones, ones_like, fill Examples -------- >>> np.array([1, 2, 3]) array([1, 2, 3]) Upcasting: >>> np.array([1, 2, 3.0]) array([ 1.,  2.,  3.]) More than one dimension: >>> np.array([[1, 2], [3, 4]]) array([[1, 2],        [3, 4]]) Minimum dimensions 2: >>> np.array([1, 2, 3], ndmin=2) array([[1, 2, 3]]) Type provided: >>> np.array([1, 2, 3], dtype=complex) array([ 1.+0.j,  2.+0.j,  3.+0.j]) Data-type consisting of more than one element: >>> x = np.array([(1,2),(3,4)],dtype=[('a','<i4'),('b','<i4')]) >>> x['a'] array([1, 3]) Creating an array from sub-classes: >>> np.array(np.mat('1 2; 3 4')) array([[1, 2],        [3, 4]]) >>> np.array(np.mat('1 2; 3 4'), subok=True) matrix([[1, 2],         [3, 4]])

bisect_left(...)
bisect_left(a, x[, lo[, hi]]) -> index Return the index where to insert item x in list a, assuming a is sorted. The return value i is such that all e in a[:i] have e < x, and all e in a[i:] have e >= x.  So if x already appears in the list, i points just before the leftmost x already there. Optional args lo (default 0) and hi (default len(a)) bound the slice of a to be searched.

bisect_right(...)
bisect(a, x[, lo[, hi]]) -> index bisect_right(a, x[, lo[, hi]]) -> index Return the index where to insert item x in list a, assuming a is sorted. The return value i is such that all e in a[:i] have e <= x, and all e in a[i:] have e > x.  So if x already appears in the list, i points just beyond the rightmost x already there Optional args lo (default 0) and hi (default len(a)) bound the slice of a to be searched.

createNewStockHistoryDirectory(whereToGetSecurityNames, extraSecurityNames, whereToPutHistories, period, years_of_history=11)
#=========================================================================================================== # Create a new stockhistory directory entry # Since yahoo fails occasionally on fetch, the routine is usually run multiple times. # Subsequent runs skip over tickers already downloaded and reattempts on those with no file in the download directory. # Runtime on 20100204 (8252 securityNames Total) was 1:24:30.370038 # Re-Runtime on 20100204 (8256 securityNames Total) was 0:11:24.455133 # Re-Re-Runtime on 20100204 (8258 securityNames) was 0:12:55.323351 # Re-Re-Re-Runtime on 20100204 (8260 securityNames) was 0:08:51.551660 # 2:40:34.771402 # extraSecurityNames lists indicies, which are not presnt in the AAII data, # E.g. createNewStockHistoryDirectory("./AAII/",["%5EXAX", "%5EDJI", "%5EIXIC", "%5ERUA"],"./stockhistory/","m",11) #===========================================================================================================

fedFormatToHistoryFormat(fedData, hpath, period, historyTickername, targetTickername)
#=========================================================================================================== # create a targetFilename in history format using dates from historyTickername and rates from fedData # Note: earliest dates in FedData come first.  latest date is last. #===========================================================================================================

getDates(timeFrame)
Return space delimited dates as a len 2 tuple of 2 - delmited strings.  Used in file name generation

listdir(...)
listdir(path) -> list_of_strings Return a list containing the names of the entries in the directory.     path: path of directory to list The list is in arbitrary order.  It does not include the special entries '.' and '..' even if they are present in the directory.

loadAllFromHmtFile(filename, period)
#=========================================================================================================== # Get security names, security histories, security ranking, from .hmt file # dbName,timeFrame,securityNames,quotes = loadFromHmtFile("TECDAX_20071203.hmt","d") # Note: securityNames lists names in order they occured in the file (ranking) # quotes[securityName] = quoteHistoryArray # "dates" are actually sample indicies #===========================================================================================================

loadFromStockhistoryDir(pathname, securityNames, earlyDate, lateDate, period, self, repair=True)
#=========================================================================================================== # Look at documents/py/autoanalyst/StockDB.py for this. # "history" directory - contains files of price/volume ticker history. # Each file contains one ticker's history.  Each filename is of the form "ticker.csv" e.g. "AA.csv" for alcoa # IMPORTANT NOTE: Make sure the first name's history covers the required interval. # Rouwenhorst does this by making the first quote an "index", e.g. %5EDJI # processed          - number of tickers passed in (usually from an exchange listed in an AAII database. # noHistories        - number of tickers deleted because finance.yahoo.com has no record of them. # pegged             - number of tickers deleted because they did not exist during the interval # corrupt            - number of tickers deleted because they have identifiable data errors (e.g. 20.0.01 as a price) # bornDuringInterval - number of tickers deleted because they did not exist at the beginning of the interval # diedDuringInterval - number of tickers deleted because they did not exist at the end of the interval # returned           - number of tickers left #===========================================================================================================

loadFromYahoo(securityNames, timeFrame, period='m')
#=========================================================================================================== # Note: Not used much anymore.  As such, does not have the recovery and reporting of loadFromStockhistoryDir # I usually just load from yahoo to stockhistory directory and load from stockhistory directory to StockDb. # E.g. myDict = loadFromYahoo(["PRLAX","PRASX","EWZ"],"2 3 1995 8 3 2009") # Months (1-12), days(1-31) and year are all origin 1 # QD TBD: code breaks if first ticker is bad #===========================================================================================================

rpt(r2, ticker)
#=========================================================================================================== #=========================================================================================================== #ABB is suspicious 0 173 0.289517 1.740433 2005-01-24 5.220000 5.160000 1.011628 2008-05-19 31.380000 31.310000 1.002236

Analyst.history index
/Users/katherinepaseman/Documents/projects/py/Analyst/history.py

#=========================================================================================================== # Copyright (c) 2006-2007 Paseman & Associates (www.paseman.com). All rights reserved. #=========================================================================================================== #=========================================================================================================== # history # - downloads ticker price histories from the internet to a file # - loads file into a dictionary # - Note that the only connection between histories and AAIIdb is the ticker symbol # File Format is: "Date" "Open" "High" "Low" "Close" "Volume" "Adj Close" # Usage: # downloadHistories(["SAP","T"]) # puts history file for "SAP" and "T" tickers in local folder # myDictionary = loadHistory("SAP") # converts molst recent downloaded "SAP" file to dictionary # p=myDictionary["2007-02-15"]["Adj Close"] # Return Adjusted close for SAP on February 15, 2007 #=========================================================================================================== # DateStampedPath File Organization # Description # The "history" directory contains "byDate" directories. # "byDate" directories contain "byPeriod" directories. # "byPeriod" directories contain histories of all ticker symbols listed in the current AAII db # A "byDate" directory name is of the format YYMMDD and records ticker histories for different periods # A "byPeriod" directory name is one of "m" month "w" week and "d" day and records ticker histories for years prior to YYMMDD # - e.g. 20070303 contains the histories of all tickers up to and including March 3, 2007. # # A "byTicker" directory contains one "tickerHistory" file for each ticker Symbol. # Each "tickerHistory" file has the name of the ticker symbol whose history it contains. # A "byTicker" directory takes about 304 megabytes on disk and takes about 2 hours to create over broadband. # # Example # stockhistory <- "stockhistory" directory. There is one "stockhistory" directory # -- 20070303 <- byDate directory There are multiple "byDate" directories # ---- d <- byPeriod directory There are at most 3 ("m", "w", "d") "byPeriod" directories # ------ A.csv <- tickerHistory file for ticker "A" on 20070303 with a daily period # ------ AA.csv <- tickerHistory file for ticker "AA" on 20070303 with a daily period # ------ ... # ---- 20070304.csv <- dailyQuote file # # # (antiquated): A "history" directory also contains "dailyQuote" files. # Each "dailyQuote" file contains the statistics for every AAII db ticker for a particular day. #=========================================================================================================== # downloadHistory - copy URL file to local file # - takes a ticker symbol, gathers daily data from last -n- years to current day, stores in file # Example: # print downloadHistory("SAP", 4,"chart.yahoo.com","history/") # returns the following header # Date Open High Low Close Volume Adj Close # # print downloadHistory("ADS.DE",4,"chart.yahoo.com","history/") # for DAX tickers. This is different from a few years ago. # Should never be used directly by end user #===========================================================================================================

Modules

os
urllib
urllib2

Functions


createTickerFileName(path, ticker, extension)
#=========================================================================================================== # Copyright (c) 2006-2007 Paseman & Associates (www.paseman.com).  All rights reserved. #=========================================================================================================== #=========================================================================================================== # history # - downloads ticker price histories from the internet to a file # - loads file into a dictionary # - Note that the only connection between histories and AAIIdb is the ticker symbol # File Format is: "Date"        "Open"  "High"  "Low"   "Close" "Volume"        "Adj Close" # Usage: # downloadHistories(["SAP","T"])            # puts history file for "SAP" and "T" tickers in local folder # myDictionary = loadHistory("SAP")         # converts molst recent downloaded "SAP" file to dictionary # p=myDictionary["2007-02-15"]["Adj Close"] # Return Adjusted close for SAP on February 15, 2007 #=========================================================================================================== # DateStampedPath File Organization # Description # The "history" directory contains "byDate" directories. # "byDate" directories contain "byPeriod" directories. # "byPeriod" directories contain histories of all ticker symbols listed in the current AAII db # A "byDate" directory name is of the format YYMMDD and records ticker histories for different periods # A "byPeriod" directory name is one of "m" month "w" week and "d" day and records ticker histories for years prior to YYMMDD # - e.g. 20070303 contains the histories of all tickers up to and including March 3, 2007. # # A "byTicker" directory contains one "tickerHistory" file for each ticker Symbol. # Each "tickerHistory" file has the name of the ticker symbol whose history it contains. # A "byTicker" directory takes about 304 megabytes on disk and takes about 2 hours to create over broadband. # # Example # stockhistory    <- "stockhistory" directory.  There is one "stockhistory" directory # -- 20070303     <- byDate directory           There are multiple "byDate" directories # ---- d          <- byPeriod directory         There are at most 3 ("m", "w", "d") "byPeriod" directories # ------ A.csv    <- tickerHistory file for ticker "A" on 20070303 with a daily period # ------ AA.csv   <- tickerHistory file for ticker "AA" on 20070303 with a daily period # ------ ... # ---- 20070304.csv <- dailyQuote file # # # (antiquated): A "history" directory also contains "dailyQuote" files. # Each "dailyQuote" file contains the statistics for every AAII db ticker for a particular day. #=========================================================================================================== # downloadHistory - copy URL file to local file # - takes a ticker symbol, gathers daily data from last -n- years to current day, stores in file # Example: # print downloadHistory("SAP",   4,"chart.yahoo.com","history/") # returns the following header # Date  Open    High    Low     Close   Volume  Adj Close # # print downloadHistory("ADS.DE",4,"chart.yahoo.com","history/") # for DAX tickers.  This is different from a few years ago. # Should never be used directly by end user #===========================================================================================================

downloadHistories(tickers, hpath='./stockhistory/', period='d', years_of_history=11, sourceSite='chart.yahoo.com')
#def downloadHistories(tickers,hpath=historyPath,period="d",years_of_history = 11): #20150608

downloadHistory(ticker, years_of_history, sourceSite, destinationPath, period='d')

downloadHistoryToString(ticker, years_of_history, sourceSite, period='d')

downloadToday(tickers)
#=========================================================================================================== # Put today's information for passed tickers in a comma delimited file # Bug: dates are in wrong format # Bug: S&P index label is wrong ("%5EGSPC" appears as ^GSP in the dowloaded file) # Bug: Some symbols are quoted (need to save as tab delimited from Excel) #===========================================================================================================

downloadTodayToString(tickers)
#=========================================================================================================== #=========================================================================================================== # Put today's information for passed tickers in a list of comma delimited strings # Todo: systemetize file formats #===========================================================================================================

getCompiledPrices(db, ticker, indicies)
#=========================================================================================================== # Used to speed inner loop of stockScreen # Must match signature of getYahooPrices #=========================================================================================================== #from StockDb import printit   # QD Comented ou on 20100204

getYahooPrices(db, ticker, times, period)

loadHistory(ticker, period)
#=========================================================================================================== # loadHistory - local file to dictionary #===========================================================================================================

loadHistoryAsList(ticker, period)
#=========================================================================================================== # loadHistory - local file to ListDict #===========================================================================================================

loadMostRecent(period)
#=========================================================================================================== # loadMostRecent - local file to dictionary #===========================================================================================================

localtime(...)
localtime([seconds]) -> (tm_year,tm_mon,tm_mday,tm_hour,tm_min,                           tm_sec,tm_wday,tm_yday,tm_isdst) Convert seconds since the Epoch to a time tuple expressing local time. When 'seconds' is not passed in, convert the current time instead.

refreshHistoryDB(db)
#=========================================================================================================== # Main External Entry Point #===========================================================================================================

Data

historyPath = './stockhistory/'

Analyst.AAII index
/Users/katherinepaseman/Documents/projects/py/Analyst/AAII.py

# -*- coding: utf-8 -*- #=========================================================================================================== # Copyright (c) 2006-2010 Paseman & Associates (www.paseman.com). All rights reserved. #=========================================================================================================== # o AAII (American Association of Individual Investors) distributes a monthly CD containing # corporate information, keyed by ticker symbol, with (as of 2006) 8,706 rows and 2,815 columns. (2009 - 9877 rows) # o The AAII class holds a -read only- dictionary of a -stringified version- of this data # - Tickers (which we refer to as securityNames) key 2,815 columns across the 14 AAII "segment" files # (F01.txt,F01_Key.txt) thru (F14.txt,F14_Key.txt). # The ticker (key) is in the first column in each file. Column names are unique across all segment files. # Each value is a string and -cannot- be coerced. (see "buggy" coerceColumn below), use copyColumn instead # - AAIIdb.idx provides a -read only- interface to this data keyed by 'securityName' and 'columnName'. # Behind the scenes, it demand loads and "joins" AAII segments. # #=========================================================================================================== # Usage: # db = AAIIdb(path) # initialize. load all path/F??_Key.TXT files # db.idx("AA","MKTCAP") # For the AA securityName, get the company market cap. As a side effect, the first call # # demand loads the segment containing the MKTCAP column for all 8,706 aaii rows. # # In all, there are 2,815 aaii columns spread across 14 segments # db.copyColumn("MKTCAP",float) # Create a dictionary, (key, value) = (securityName, indicated column) # # The indicated column is coerced #=========================================================================================================== # self.columns[columnName] = segmentId,segmentFileName,offset # self.rowIds[securityName] = rowId #=========================================================================================================== # Useful columns: (See 20040206 Column Table.xls for a complete list as of 2004) # MKTCAP - Market capitalization in millions # EXCHG_DESC - Exchange Description - "A - American", "M - Nasdaq", "N - New York", "O - Over the counter" # COMPANY - Company name # # MULTIPLES # PE Price/Earnings per Share # PBVPS Price/Book Value per Share # PSPS Price/Sales per Share # PCFPS Price/Cash Flow per Share # PFCPS Price/Free Cash Flow per Share # # PROFITABILITY # Gross Profit Margin # Operating Margin # Net Profit Margin # Return on Assets # Return on Equity # # LIQUIDITY # Quick Ratio # # PRICE # PRICE_M001 - PRICE_M120 - The closing price of the last trading day for each of the last 120 months. # PRICEDM001 - The date of the last trading day for each of the last 120 months. # PRICEHM001 - The highest price the stock has traded at in each of the last 120 months. # PRICELM001 - The lowest price the stock has traded at in each of the last 120 months. # PRICEVM001 - The total trading volume for a company’s stock for each of the last 120 months. #===========================================================================================================

Modules

glob

Classes

AAII

class AAII

#===========================================================================================================

Methods defined here:

__init__(self, pathname, preload=False)

coerceColumn(self, columnName, coercionType)

copyColumn(self, columnName, coercionType)

copyColumns(self, columnNames, coercionType)

getNamesQuotesDates(self, securityNames, earlyDate, lateDate, period)

getSecurityNames(self, triggerColumn='COMPANY')

idx(self, securityName, columnName)

Data
		Rpipelines = [{'name': 'plR_m2', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(2); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 2 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m3', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(3); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 3 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m4', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(4); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 4 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m5', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(5); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 5 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m6', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(6); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 6 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m7', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(7); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 7 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m8', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(8); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 8 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m9', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(9); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, 9 members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}] i = 9 plRm = 'computeSecurityReturns(); computeRouwenhorstSecu...erCount(%d); computePortfolioReturns(); report();' shortTimeFrames = ['1 4 2015 12 15 2016']

Data
		Mpl = "computeSecurityReturns(); computeSecurityVolatil...ePortfolioReturns(); dump(); report('WinnerOnly')" exp = <ufunc 'exp'> inf = inf log = <ufunc 'log'> loserPortfolioIdx = -1 mgrid = <numpy.lib.index_tricks.nd_grid object> mpt = <Analyst.MPT.MPT.MPT instance> pipelines = [{'name': 'plR_p3', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...lioCount(3); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, Three portfolios, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_p10', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...ioCount(10); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, Ten portfolios, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m3', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(3); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, Three members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plR_m10', 'stages': 'computeSecurityReturns(); computeRouwenhorstSecu...erCount(10); computePortfolioReturns(); report();', 'text description': "Rouwenhorst ranking, Ten members, 'Winner/Loser' reporting - Rouwenhorst JOF 199802"}, {'name': 'plCC_p10', 'stages': 'computeSecurityReturns(); computeSecurityVolatil...ioCount(10); computePortfolioReturns(); report();', 'text description': "Constant Correlation ranking, Ten portfolios, 'Winner/Loser' reporting - Elton/Gruber 6thEd p195"}, {'name': 'plCC_XlsW0R40', 'stages': 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();', 'text description': 'Constant Correlation ranking, Two portfolios, Rh... reporting - Elton/Gruber 6thEd p195 j volatility'}, {'name': 'plCC_XlsW0R50', 'stages': 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();', 'text description': 'Constant Correlation ranking, Two portfolios, Rh... reporting - Elton/Gruber 6thEd p195 j volatility'}, {'name': 'plCC_XlsW0R60', 'stages': 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();', 'text description': 'Constant Correlation ranking, Two portfolios, Rh... reporting - Elton/Gruber 6thEd p195 j volatility'}, {'name': 'plCC_XlsW260R40', 'stages': 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();', 'text description': 'Constant Correlation ranking, Two portfolios, Rh...orting - Elton/Gruber 6thEd p195 12 mo volatility'}, {'name': 'plCC_XlsW260R50', 'stages': 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();', 'text description': 'Constant Correlation ranking, Two portfolios, Rh...orting - Elton/Gruber 6thEd p195 12 mo volatility'}, {'name': 'plCC_XlsW260R60', 'stages': 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();', 'text description': 'Constant Correlation ranking, Two portfolios, Rh...orting - Elton/Gruber 6thEd p195 12 mo volatility'}, {'name': 'plCC_XlW0R40', 'stages': "computeSecurityReturns(); computeSecurityVolatil... computePortfolioReturns(); report('WinnerOnly');", 'text description': "Constant Correlation ranking, C* portfolio forma...inner Only' reporting - Elton/Gruber 6thEd p195-6"}, {'name': 'plCC_XlW0R50', 'stages': "computeSecurityReturns(); computeSecurityVolatil... computePortfolioReturns(); report('WinnerOnly');", 'text description': "Constant Correlation ranking, C* portfolio forma...inner Only' reporting - Elton/Gruber 6thEd p195-6"}, {'name': 'plCC_XlW0R60', 'stages': "computeSecurityReturns(); computeSecurityVolatil... computePortfolioReturns(); report('WinnerOnly');", 'text description': "Constant Correlation ranking, C* portfolio forma...inner Only' reporting - Elton/Gruber 6thEd p195-6"}] plCC = 'computeSecurityReturns(); computeSecurityVolatil...teCCX%s(); computePortfolioReturns(); report(%s);' plCC_XlW0R40 = "computeSecurityReturns(); computeSecurityVolatil... computePortfolioReturns(); report('WinnerOnly');" plCC_XlW0R50 = "computeSecurityReturns(); computeSecurityVolatil... computePortfolioReturns(); report('WinnerOnly');" plCC_XlW0R60 = "computeSecurityReturns(); computeSecurityVolatil... computePortfolioReturns(); report('WinnerOnly');" plCC_XlsW0R40 = 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();' plCC_XlsW0R50 = 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();' plCC_XlsW0R60 = 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();' plCC_XlsW260R40 = 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();' plCC_XlsW260R50 = 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();' plCC_XlsW260R60 = 'computeSecurityReturns(); computeSecurityVolatil...ongShorts(); computePortfolioReturns(); report();' plCC_p10 = 'computeSecurityReturns(); computeSecurityVolatil...ioCount(10); computePortfolioReturns(); report();' plR = 'computeSecurityReturns(); computeRouwenhorstSecu...ioCount(%d); computePortfolioReturns(); report();' plRR_m10 = 'Rouwenhorst(10); report();' plRR_m3 = 'Rouwenhorst(3); report();' plRRm = 'Rouwenhorst(%d); report();' plR_m10 = 'computeSecurityReturns(); computeRouwenhorstSecu...erCount(10); computePortfolioReturns(); report();' plR_m3 = 'computeSecurityReturns(); computeRouwenhorstSecu...berCount(3); computePortfolioReturns(); report();' plR_p10 = 'computeSecurityReturns(); computeRouwenhorstSecu...ioCount(10); computePortfolioReturns(); report();' plR_p3 = 'computeSecurityReturns(); computeRouwenhorstSecu...lioCount(3); computePortfolioReturns(); report();' plRm = 'computeSecurityReturns(); computeRouwenhorstSecu...erCount(%d); computePortfolioReturns(); report();' template = '<html>\n<head>\n<title>%s %s</title>\n\n<script type...1>%s %s</h1><br><h2>%s</h2><br>%s</body>\n</html>\n' winnerPortfolioIdx = 0

Data
		inf = inf loserPortfolioIdx = -1 mgrid = <numpy.lib.index_tricks.nd_grid object> sqrt = <ufunc 'sqrt'> winnerPortfolioIdx = 0

Data
		loserPortfolioIdx = -1 winnerPortfolioIdx = 0

Data
		abs = <ufunc 'absolute'> log = <ufunc 'log'> loserPortfolioIdx = -1 maxBias = 21.0 minimum = <ufunc 'minimum'> oneOverTauTable = {'d': 252.0, 'm': 12.0, 'w': 52.0, 'y': 1.0} sqrt = <ufunc 'sqrt'> winnerPortfolioIdx = 0

Data
		DAXcomponents = ['ADS.DE', 'ALV.DE', 'BAS.DE', 'BAYN.DE', 'BEI.DE', 'BMW.DE', 'CBK.DE', 'CON.DE', 'DAI.DE', 'DB1.DE', 'DBK.DE', 'DPW.DE', 'DTE.DE', 'EOAN.DE', 'FME.DE', 'FRE.DE', 'HEI.DE', 'HEN3.DE', 'IFX.DE', 'LHA.DE', ...]