Category Archives: Uncategorized

How to download legacy electricity market data from SEM-O

The Irish Single Electricity Market Operator’s website changed when the new I-SEM electricity market commenced in 2018. However, it is still possible to download historic data from the pre-2018 SEM. A previous post discussed how to obtain the output of individual generator units on the SEM. This post describes how to download historic market information. This includes wind forecasts, wind generation, generator outputs and prices. Here, the instructions to download market prices are given.

Go to the legacy SEM-O page:

http://lg.sem-o.com/pages/default.aspx

Sign in.

Choose:

Market Data > Dynamic Reports

Report Type  : Dynamic Reporting – Tables

Report Group : All

Report: Market Results (System Load SMP and Shadow Price)

In the following screen, choose run ‘EA2’ (ex-Ante run 2)

From : date in mm/dd/yyyy To: mm/dd/yyyy

Click ‘popout report’

  • The report can be downloaded in CSV form and viewed in a text editor or Microsoft Excel.

The data can be interpreted as follows:

SMP is the ‘system marginal price’ in Euro cent. The data is presented on half hourly intervals. Each ‘Delivery Hour’ has two ‘delivery intervals’. Delivery interval 1 for hour X runs from X:00 to X:30 and delivery interval 2 runs from X:30 to [X+1:00].

Wind data sources

A collection of wind data sources in the public domain.

winddata.com – huge array of wind resource data from Denmark and other countries, compiled by DTU. Some wind turbine generation data and structural response data also. Registration is required and not all data is free.

Egmond aan Zee offshore wind farm data — c. four years of meteo data and other useful information  from a wind farm site off the Dutch coast.

FINO 1,2,3 masts in the North Sea — meteo data from three different sites available for several years. Registration required. Database here.

Wind data discoverable via OpenEI – a wide range of wind maps, wind resource data for many countries

Bonneville Power Administration (USA) – transmission data, wind generation, meteo data, historic wind forecasts

Eirgrid – historic total wind generation, interconnection flows, forecasts and other transmission system level data from Ireland. Open access, no registration required.

Irish wind generation, market and forecast data also available from SEMO , see post here for details. Registration required.

 

 

 

PhD Studentship in Peatland Carbon Cycling and Greenhouse Gas Fluxes

Duration: 3 Years

Funded at €23770 per annum from which University fees (currently €5770 per annum for EU students) are payable.

Start date: Negotiable.

Location: Cork, Ireland

Project description

Many of Ireland’s raised bogs were drained and subjected to small-scale and industrial peat extraction in the 20th century.

Today, there is a move to remediate large areas of cutaway peatlands through rewetting and allowing natural revegetation. However, there is an urgent need for scientific evidence to identify the optimal management strategies for these sites in order to minimise current and future emissions of greenhouse gases, particularly carbon dioxide and methane.

This project will involve the collection and analysis of continuous GHG flux measurements from an eddy covariance tower which has been operating at a cutaway bog site in the midlands of Ireland since 2016. Peat extraction at this site ceased two decades ago and the site has been colonised by birch and scots pine. This peatland therefore represents a huge opportunity to identify sustainable land use and management practices for the future of Ireland’s cutaway raised bogs.

The student will also be expected to quantify other components of the carbon and water balance of the peatland, via ecological/biometric surveys, static chamber measurements to constrain fluxes from particular  areas, eco-hydrological fieldwork and empirical modelling of GHG fluxes.

Essential Criteria

An interest in carbon cycling and greenhouse gas dynamics of terrestrial ecosystems and a willingness to work on field experiments, across all seasons, in a peatland forest. The successful candidate should be technically minded, highly motivated, prepared for extensive field based and laboratory research, have good data analysis and written and oral communication skills, while a full clean EU driving license and fluency in English are essential. A 2.1 Honours (or equivalent) Bachelor’s Degree, or a Master’s Degree in one of the following: Plant Ecology, Botany, Forestry / Agricultural Science, Physics / Engineering or another subject, provided the candidate has relevant research experience

Prior experience in micrometeorology, eddy covariance techniques or the measurement of greenhouse gases are advantageous.

The project is supported financially by Bord na Mona and the scientific supervision team includes Dr Paul Leahy (UCC), Dr Matthew Saunders (Trinity College Dublin), Dr Catherine Farrell (Bord na Mona) and Dr Matteo Sottocornola (Waterford Institute of Technology). The project will also be supported logistically by the Lullymore Heritage Park.

To apply

Please send a CV and a cover letter by email to :

Dr. Paul Leahy, Email: paul.leahy@ucc.ie | tel +353 21 4902017

School of Engineering, University College Cork, Cork, Ireland.

 

More on TIGGE-LAM forecasts

This is a prequel to the previous post on Extracting point forecasts from TIGGE-LAM grib archives. 

How do you get access to the TIGGE-LAM data, and once you do have access, how do you know which data to download?

The following is a short guide:

  1. Register as a user with ECMWF.   Visit https://www.ecmwf.int/, click on login, then on ‘Register’.
  2. Your registration may take a day or two to be approved.
  3. Login, then visit the TIGGE-LAM data portal
  4. Read the important notice on the MARS server’s efficiency before attempting to download data: https://screenshots.firefox.com/1T73Hh78xsuAYpNw/apps.ecmwf.int
  5. Assuming you just want a point forecast and not a whole ensemble, select ‘prod‘ (production) as the version, select ‘control forecast
  6. Select the time period (months and years) you are interested in.
  7. Select the model from the TIGGE-LAM family whose output you wish to download. We used HIRLAM from the Danish Meteorological Institute because of its relatively high update frequency (every six hours).
  8. Select the parameters of interest, e.g. 10 m u and v components of wind speed.
  9. Submit the request
  10. It may take several hours or even days for the request to be processed.
  11. Check your job’s progress on http://apps.ecmwf.int/webmars/joblist/

The ECMWF data portal also offers you a choice of output format. In the previous post on TIGGE-LAM I described how to deal with the default rotated lat/lon grid. However, other output format options are offered which may be more useful for your particular application.

Postdoc opportunity – sustainable models for the reuse of wind turbine blades

We are seeking a postdoctoral research assistant to work on a new project, jointly funded by Science Foundation Ireland, the National Science Foundation of the USA, and the Department of Enterprise and Learning (Northern Ireland).

With many wind turbines in Ireland approaching the end of their service lives, the issue of the  fate of composite wind turbine blades is coming to the fore.  Composite materials are not easily recycled, therefore such blades are typically landfilled, incinerated or set aside when turbines reach the end of their service lives.

The postdoc will develop a methodology for engaging communities and other key stakeholders in decisions regarding novel re-uses of end-of-life (EOL) wind turbine blades. The aim of the methodology will be to devise sustainability models, including in-depth conceptualisation of the monetary and non-monetary value generation arising from various reuse alternatives, thus increasing the confidence of investors and other key stakeholders. Such business models will include collaborative approaches where the researchers work with identified stakeholder organizations.

A systematic framework for performing an inclusive life cycle assessment (LCA) on the commercial software platform, SimaPro, using life–cycle inventories (LCIs) fromEcoInvent and NREL, and Life–Cycle–Impact–Assessment (LCIA) criteria (e.g., the EPA’s TRACI 2.0) will be created and tested in the Irish context. This will be used not only to determine sustainable options (environmentally, economically, and socially) for reuse of FRP blades but also to optimize the parameters to maximize sustainability. Methods and tools for the co-creation of viable, sustainable and community enhancing business models for the reuse of EOL wind turbine blades will be established.

The post duration is 12 months from 1st April 2018.

For further details please contact Paul Leahy, paul.leahy@ucc.ie.

The official vacancy is published, with details of how to apply, on UCC’s Academic Vacancies site.

Extracting point forecasts from TIGGE-LAM grib archives

The European Centre for Medium Range Weather Forecasting (ECMWF) provides an archive of high-resolution forecasts generated by several limited-area numerical weather prediction models, as part of the TIGGE-LAM (The International Grand Global Ensemble – Limited Area Model) project. One of these forecast datasets is generated by HIRLAM, run by the Danish Meteorological Institute . The TIGGE-LAM datasets are freely available for research purposes from ECMWF.

For wind energy forecasting studies, the TIGGE-LAM datasets are preferable to more well-known numerical weather prediction datasets such as ERA-Interim because TIGGE-LAM contains pure forecasts at multiple lead times, rather than reanalysis data. The HIRLAM forecasts of TIGGE-LAM also have very high horizontal resolution (approximately 0.05 degrees in latitude and longitude). The wider TIGGE project is described in detail in [1].

The HIRLAM data is provided in GRIB format on a rotated lat/lon grid (south pole co-ordinate 40 degrees S, 10 degrees E). If you want to extract data for a particular point location you have to either (a)  transform the dataset to an “unrotated” grid, or (b) transform your particular co-ordinates to the HIRLAM grid. The second option, (b) seemed more expedient, and I found a very useful Matlab script, rotated_grid_transform() , by Simon Funder to do this.

Once the location was transformed to HIRLAM’s rotated grid co-ordinates  (and reverse transformed to check everything worked properly!) ,  I ran grib_ls with the “-l” command line option to extract the four nearest neighbouring grid cells’ values from the grib file.

Some other points to consider are:

(1) HIRLAM’s grid is Arakawa-C, so the grids for u and v are staggered.  I think this means that u and v should be separately spatially interpolated to the point of interest, but the spatial resolution of HIRLAM is good, so that for now I just took the nearest grid cell of each field (u and v).

(2) the wind direction calculated from u and v will be rotated because the native HIRLAM grid is rotated. Again, I’m not (currently) interested in wind directions (just wind speeds) so I didn’t try to “unrotate”  this .

[1] Swinbank, R.; Kyouda, M.; Buchanan, P.; Froude, L.; Hamill, T. M.; Hewson, T. D.; Keller, J. H.; Matsueda, M.; Methven, J.; Pappenberger, F.; Scheuerer, M.; Titley, H. A.; Wilson, L. & Yamaguchi, M. The TIGGE Project and Its Achievements.
Bulletin of the American Meteorological Society, 2016, 97, 49 – 67

Obtaining historic generator production data from SEM-O

The Single Electricity Market Operator for Ireland publishes a lot of useful data on its portal. You can view current and historic wholesale electricity prices, aggregate wind generation and wind forecasts, etc. Not only does SEM-O provide data, but it also provides a web interface which is very user-friendly, once you understand some of the Single Electricity Market’s terminology.
You can use SEM-O’s data browsing tools to view the past output, i.e. metered generation, of any individual generator unit on the Single Electricity Market.

Here’s an example:

1. First, register for an account with SEM-O.

2. Next, examine the list of generator unit ids to find the one you are interested in. Note that many power plants operate as more than one generator unit, so if you would like to know the total generation output of a plant, you need to know all the generator units associated with that plant. The generator unit id is a string field in the format “GU_xxxxxx” where xxxxxx is a six-digit number.

3. Now go to the Dynamic Reporting tool.

SEM-O dynamic report generator tool. SEM-O website is copyright 2017 SEMO, ROI: The Oval, 160 Shelbourne Road, Ballsbridge, Dublin 4.

4. Select Report Type: “Tables”; Report Group “Energy Data”; Report “Metered Generation by Unit”.

5. There are other fields you can use to filter your search to particular time ranges (“delivery days” in SEM-O terminology) in the format dd/mm/yyyy.

6. For run type select “EP2” (ex-post run 2).

7. One filter you need is the “Unit Name” — here you enter the generator unit id for the particular unit you are interested in.

8. Click on ‘pop-out report’ (make sure that your browser allows pop-up windows from sem-o.com).

9. You can now view the data on-screen, or download a copy of your requested data. If the report is empty, then something has gone wrong. Check that your delivery date ranges and your generator unit value are valid.

10. Select a format to export (useful predefined formats offered include CSV, Excel and PDF) and click on the “Export” link.

SEM-O dynamic report pop-up window example. SEM-O website is copyright 2017 SEMO, ROI: The Oval, 160 Shelbourne Road, Ballsbridge, Dublin 4.

11. Interpreting the data columns: the generator unit is in the column headed “TYPE PART UNIT”. The Run Type (EP1 or EP2) is also recorded in a column. The delivery date refers to the date the energy was “delivered” from the unit to the market. Each hour in the SEM is partitioned into two half-hour delivery intervals, indicated as 1 and 2 in the “Delivery Interval” column. The final column, headed “MG” is metered generation. This is provided in energy units (i.e. MWh) so don’t forget to divide by the time interval (0.5 hour) if you would like to know the mean power output for the interval.

GRIB command line tools from ECMWF

ECMWF supplies some useful command line tools for handling GRIB files: grib_get, grib_ls and grib_dump and several others. The tools are supplied as part of the latest version of the GRIB API.

To use the tools, you have to install the GRIB API from ECMWF:
https://software.ecmwf.int/wiki/display/GRIB/GRIB+API+CMake+installation

Recent versions weren’t available from any of the remote repositories in my system’s default list. Therefore I had to download the installation package of a recent version of the grib-api (v. 1.14.0 or later) manually and then install it from a local repository using the Synaptic package manager in Ubuntu. See here for details of how to do this. Also, make sure you know whether your system is 32-bit or 64-bit and download the appropriate package. For my 32-bit Ubuntu system I used the 32-bit Debian package from the ECMWF GRIB-API Releases list.

With the API installed I was able to examine the contents of the files using grib_dump. After figuring out what was inside the files, I then extracted a time series for a specific location (specified as a latitude,longitude pair) using :

#!/bin/sh
# u component, all validity times
grib_ls -w shortName:s=”u”,stepRange=3 -l 51.8483,-8.483543 -p dataDate,dataTime,validityDate,validityTime MERA_PRODYEAR_2015_06_33_105_10_0_FC3hr.grb > test33.txt
# v component, all validity times
grib_ls -l 51.8483,-8.483543 -p dataDate,dataTime,validityDate,validityTime -w shortName:s=”v” MERA_PRODYEAR_2015_06_34_105_10_0_FC3hr.grb > test34.txt

This extracted values for the four nearest grid cells to the specified location, for all forecast validity times (1h, 2h, 3h). There is a ‘grib_filter’ tool also provided which looks very useful as it can be used to match data based on a set of rules specified in a separate file. This looks better than the approach I took, but for now was just happy to get some data out of the GRIB files.

Handling data in Lambert Conformal Conic projections

I want to extract a time series of wind forecast values for a specific location from a GRIB file issued by Met Eireann as part of the MERA reanalysis project. In a previous post I listed several software tools for interrogating GRIB files. For various reasons, I wasn’t able to persuade any of those tools to do what I needed.

NASA GISS’ Panoply viewer was useful for visualisation, and it does have a facility for bulk export of data to CSV files. However, as the files are exported as “flat” CSV, they are very large. Furthermore, without knowing the details of how the CSV files are structured, the file layout is difficult to interpret. There is the additional problem of georeferencing the exported data to given latitude/longitude pairs, as the values are exported on the model’s Easting-Northing grid.

Before going further, some details of the MERA dataset are relevant: the projection is Lambert Conic Conformal. The GRIB file format is GRIB-1. The data are provided on an x-y (Easting-Northing) grid.

I wrote some Matlab routines based on the formulae provided in [1] and [2] to calculate latitudes and longitudes for the MERA grid cells for the Lambert Conformal Conic projection (code available on github). I could use those to find the co-ordinates of MERA grid cells corresponding to the latitude and longitude of particular locations of interest. But I still wasn’t able to extract the data from the GRIB file even when the grid cell co-ordinates were known. CDO appears to have difficulty with Lambert Conformal Conic projections. The NCL tools could not be persuaded to extract data.

In the end, following a tip from Met Eireann, I turned to the GRIB_API provided by ECMWF. Some useful command-line tools are provided with the API package and I was able to use these to query and “slice and dice” the MERA GRIB files. I’ll provide the details in another post.

[1] European Petroleum Survey Group. Guidance Note Number 7 on “POSC literature pertaining to Coordinate Conversions and Transformations including Formulas”, pp. 17-18.
[2] US Geological Survey Professional Paper 1394, “Map projections -a working manual”. Snyder, J. P. (1987) pp. 107-108.

Surface wind speeds and directions from MERA

Previous posts have discussed software tools for extracting and visualising data from GRIB archives. With some help, I’ve used Panoply to read and visualise surface wind speeds from the sample Met Eireann Reanalysis (MERA) data files.

Separate GRIB archives of the u- and v- components of the horizontal surface (10 m) wind velocity are provided. Panoply can import multiple datasets and it can also be used to combine the u- and v- components in order to visualise surface wind speeds and directions. This page from NASA’s Goddard’s Earth Science Data and Information Services Center describes how to do this.

Here’s an example of the output from 03:00, 1st June 2015, with prediction intervals of 1, 2 and 3 hours respectively:


10 m surface wind speeds (m/s) and vectors from MERA from 03:00 the 1st June 2015, with prediction intervals of 1, 2 and 3 hours.

I’m still unsure of the timestamp convention. For each of these images, Panoply reported the “GRIB reference time” as 2015-06-01 15:00+0000. The corresponding “GRIB Forecast or Observation Time” entries are 2015-06-01 01:00+0000, 2015-06-01 02:00+0000 and 2015-06-01 03:00+0000. This is a bit confusing because the Forecast times appear to be before the reference time but I think it may just be a glitch in either the GRIB file or in how it is handled by Panoply. My interpretation is that these charts correspond to the model run for 2015-06-01 15:00 for prediction intervals of 1h, 2h and 3h, i.e. for 16:00, 17:00 and 18:00 on the same day as the model run. Met Eireann’s paper from the European Meteorological Society Conference states that 3 hours of predictions are supplied for each model run, apart from the 00:00Z run.

Next step: to ‘slice and dice’ these data in order to get time series of wind speeds for a particular location. Panoply seems to be able to do this, but wgrib might be a bit more practical when it comes to larger datasets.